JP Moresmau's Programming Blog

Lego Mindstorms, Haskell, Windows: all set!

2012-01-16T23:06:00.000+01:00

A new version of the NXT library has been released, and it supports Windows!! I lamented a few weeks ago that it only ran on Unix/MacOs (requiring the unix package and using Posix file descriptors), but that's a thing of the past. It was really easy to port Mitar's code to use the serialport package instead of file descriptors. Now Mitar has merged back my changes, we've tested the code on Linux and Windows, and it's been released on Hackage!

So now I can control my Lego Mindstorm robot from Haskell on my Windows machine! World domination is only a couple more steps away!

I wish to point out that Mitar's code is extremely well commented and a pleasure to work with, which of course made the adaptation to Windows a lot easier. Thank you!

EclipseFP 2.2.1 released

2012-01-13T14:55:00.000+01:00

It is my pleasure to announce a new version of EclipseFP. Release 2.2.1 does not bring huge changes in the UI, but comes with important bug fixes and small enhancements. More importantly, it integrates with the latest versions of buildwrapper and scion-browser. You can see the release notes here.

We tried to enhance the performance and stability of EclipseFP, while making it easier to work with. The "open definition" action for example should be able to resolve a lot more symbols than before, including in type annotations. Quick fixes and auto-completion we've also worked on.

I wish to thank all users of EclipseFP, most notably all the people that submit bug reports or questions to the forum and thus help make the product move forward!

The instructions to install are the same as always: point your Eclipse to the update site http://eclipsefp.sourceforge.net/updates/. Once you restart EclipseFP should install or update buildwrapper and scion-browser.

Happy Haskell Hacking!

EclipseFP and "the future of programming"

2011-12-30T17:30:00.000+01:00

Paul Chiusano's essay "The Future Of Programming" has sparkled a lot of debate, for example on Reddit. His points on IDEs remind me somewhat on what Luke Palmer has to say on them. And he's strongly in favor of lazy, statically-typed, functional languages.

I suddenly find myself in a strange (good?) position: as a maintainer of an IDE (ok, plugins) for Haskell, I can somehow participate towards fulfilling that dream. I know, I know, we're far from it, and EclipseFP has not yet reached the ease of use and power of the JDT in Eclipse. But maybe we can achieve success by doing things differently, not by trying to mimic what's already there and anyway applies to a different language.

We've already started to take some steps in the right direction: for example, Alejandro is working on a version of scion-browser that stores its data in a database. It's not such a big leap to envision that a lot of information about the developer's projects and modules could be kept in that database, to give accurate and fast searching and refactoring operations. We'll also need to think about providing an editor that is a lot more structured than the standard text-with-foldable-sections editor we have now, but it's not clear now if

the benefits of a structured editor justify developing an interface programmers are not used to
people would take the plunge even if we could prove tangible benefits

I can already see a clear advantage in having an editor that always force you to have valid code: EclipseFP loses a lot of functionality (thingAtPoint for example doesn't work) if the module you're working on has errors, because the GHC API can provide a lot more info once the module can be analysed correctly. With things like "undefined" and type-directed autocomplete support we could have modules that are always correct. I'm not sure myself that having a compiling module with loads of undefined that fails miserably at run-time is such a great proposition, but hey... In Java, Eclipse let you run a file with errors and only throws a run-time error when you access the incorrect code.

So we'll have to overcome usability and performance issues, deal with GHC limitations and programmers' expectations, but we're in the position to at least try, and see if we can move forward a little bit!

Lego Mindstorms and Haskell?

2011-12-28T21:41:00.000+01:00

Santa brought me a Lego Mindstorms kit! Suddenly the time I'm going to have available for EclipseFP is greatly reduced (I joke). I'm having fun building my first little models and fiddling with the graphical interface for programming the little beasts, but I was wondering if I could use Haskell to control the robots. Things look bleak:
- the NXT library on Hackage looks good, but requires unix... Maybe getting it to work on Windows would be easy, but I wonder...
- the fantom development kit (fantom is the low level communication API for Mindstorms, if I understand correctly) comes with a Windows version that will only work with Visual C++. It uses C++ lib and dll files whose name mangling is not compatible with MinGW.

So as a Windows and MinGW user, I'm in a bit of a ditch. I'm wondering now about the Urbi platform, that seems to be open source and to have a module for Mindstorms. It seems to provide a C++, Java and scripting interfaces, so it may be possible to get a Haskell wrapper on top of it. I wonder has anybody worked on Haskell bindings for Urbi? Might be interesting...

Buildwrapper 0.2.2 released

2011-12-26T17:29:00.001+01:00

Hello all, I've just released a new version of buildwrapper, which is one of the backend Haskell executables used in EclipseFP. This is a bug fixing release, addressing some of the issues reported by people. Just install the new version via cabal install or via the EclipseFP cabal packages views.
There is ongoing work on EclipseFP and scion-browser, so a new version of both should hit the shelves pretty soon, in a matter of weeks!

Happy Haskell Hacking!

EclipseFP helper executables for Windows

2011-11-12T23:03:00.001+01:00

Since people had sometimes problems building Haskell executables we use in EclipseFP, I thought I'd help by putting versions of the current Windows executables on sourceforge. So in you run Windows and have trouble building scion-browser or buildwrapper, you can grab them from here.

I know this isn't really in the spirit of "build from sources so you can check them yaddi yadda", but since even Cabal does it...

As usual, use at your own risk, etc.

Happy Haskell Hacking!

EclipseFP 2.2.0 released!

2011-11-11T17:59:00.001+01:00

I'm glad to announce that EclipseFP 2.2.0 has been released. EclipseFP is a set of Eclipse plugins for Haskell development.

The minor version increase is justified by the replacement of the Scion library for the backend by the BuildWrapper executable. I appreciate that lots of work went and is still going into Scion, but I decided that building my own Haskell backend was better for EclipseFP. It calls cabal directly for building, which allows EclipseFP to support things like new style component dependencies (the executable in a project depends on its own library), c files (now my HJVM project, which call Java from Haskell can build, and even report errors in the C code), etc. Note the the haskell code is not bundled with EclipseFP anymore but gets downloaded and installed from Hackage.

This version also comes with a new view, allowing the installation of cabal packages from Hackage. There are also bug fixes and small enhancements all round.

To install, just use the built-in Eclipse install/update functionality. The update site URL is http://eclipsefp.sourceforge.net/updates. The release notes can be found here.

Happy Haskell Hacking with EclipseFP!

GUI Testing using OCR: project Sikuli

2011-10-20T19:44:00.000+02:00

I had a great idea this morning... and then realized that there was an open source project already doing it.
I thought: instead of using specific testing frameworks for each UI technology I have (a tool for the web, a tool for SWT UIs, etc) what about a tool that could recognize the text on screen and click or type in the proper places. The same technology for everything type of UI, and something that doesn't rely on pixel perfect positioning to work! Turns out (surprise!) I'm not the first one that had the idea, and looking around I stumbled on Project Sikuli. This looks great: it uses images to find "interesting" parts of your UI and there's also support for text recognition. So potentially you could say "click on the button that says 'Submit'"! It integrates with Java code so you can easily have JUnit tests using it.
Unfortunately, for my own purposes I couldn't get very far easily because of lack of support for transparent images (or, more precisely, it takes into account transparent areas for images, so it doesn't recognize an image if the background changes, which happens if your desktop theme changes, if you use the same icon in different contexts in your UI, etc). But it still looks like a promising tool!

Manage Cabal packages from inside EclipseFP

2011-10-13T19:51:00.000+02:00

I'm thinking the new version of EclipseFP will not bundle the Haskell code we use, but will download it from Hackage, which would make sense: we would be able to update the Haskell side without requiring a EclipseFP upgrade, and vice versa. I would expect Haskell developers would be quite used to install packages from Hackage using cabal install. However, let's try to minimize the number of things that a user would need to go outside of EclipseFP to do on a normal Haskell project. So I've added another view:

This view lets you see the list of packages you've installed, with their version, or the full list from Hackage, that you can filter by typing the prefix of the name you're looking for. From there you can install the packages as either Global or User, see the package information, or click on the "More Information" link to open the Hackage page directly.
You can also update the list. This GUI is only a wrapper above the cabal list and cabal info commands.
Hopefully this will be useful to people! Should I add a text field somewhere to be able to specify more options to cabal install, or is that overkill for a GUI?

New backend for EclipseFP

2011-10-07T14:40:00.001+02:00

A while ago I was wondering how I could enhance EclipseFP, regarding things like memory usage, support for multi components projects, etc. I now have a working version for adventurous testers!

The main thing is the removal of Scion. EclipseFP now uses another backend component called "buildwrapper". There is no server, buildwrapper is only an executable that can be launched with flags, and writes a JSON result on the standard output. This means no more huge memory usage, and less synchronization headaches.

Buildwrapper uses a copy of the project inside a hidden folder. This means that when we want to get say build errors on a dirty editor content, we can write the dirty content to the file inside the hidden folder, and launch a build process on these files, thus avoiding the limitations that the GHC API has on building from a StringBuffer (doesn't work for files requiring preprocessing, etc).

We only use the GHC API to get the types of things in a source file, but we use Cabal to build the project. Buildwrapper actually launches the cabal executable and parses the results. I know, sounds crude compare to an API only approach, but I have the feeling it just works better. Basically the IDE will now behave a lot close to what you get if you were building your project from the command line. And then you get additional goodies, like multi component support (so you can have the executable of your project reference the library in the same project, and it works!).

Buildwrapper uses haskell-src-exts for outline. I've actually seen cases where it needed language pragmas to parse correctly that GHC didn't need, but such is life.

Performance seems to be quite good, except for things that use the GHC API, where there's a little lag. Hopefully the fact that we don't use gigs of memory any more will make up for it.
I'm also thinking about distribution. EclipseFP came with Scion and Scion-browser bundled, if only because we used a different version of scion than the one on Hackage. I think that moving forward we'll rely on cabal install and Haskage. I'm going to add a simple interface to run cabal install inside EclipseFP, so you shouldn't have to go outside to install a package. So the EclipseFP preferences will just ask the user for the path to the buildwrapper and scion-browser executables, with an option to install them from hackage.

If people want to start testing, just go on github, and grab the BuildWrapper repository (main branch) and the EclipseFP project (buildwrapper branch). For the moment, BuildWrapper probably only builds on GHC 7.0. Once I have the feeling it's robust enough and there's no big flaw in the new architecture, I can work on enhancing portability.

EclipseFP 2.1.0 released!

2011-09-02T16:56:00.002+02:00

It's been a little while, but finally a new version of EclipseFP has been released! This version is mainly the work of Alejandro Serrano, who has worked on EclipseFP (and the underlying utilities) during his GSoC project. There are lots of new features in that version, mainly :

A package/module browser
Hoogle integration
Support for running tests
HLint warnings
Support for running executables and get profiling graphs
SourceGraph generation
Better auto completion information

Lots, I'm telling you. There are also fixes and internal enhancements that hopefully should make your life easier.
You can just run the update from your Eclipse IDE. You can find the full release notes there.
Note that Alejandro has also created a new web site for EclipseFP, that you can find there. Alejandro, big thanks for your hard work and your dedication!

Any feedback is welcome, and of course bug reports and requests for enhancements!

Happy Haskell hacking in Eclipse!

Hoogle Command Line on Windows

2011-08-31T17:46:00.000+02:00

The new upcoming version of EclipseFP will feature Hoogle integration, thanks to the work of Alejandro Serrano, so I was doing some testing on one of my Windows machine (yes, I'm a Haskell hacker and a Windows user, so what?)... No real problem using Hoogle via the command line, but since it does require some other things to be installed, I though I'd post them there for reference.

Unlike Cabal, Hoogle uses external tools for some of its operations. These tools are I'd say part of any self-respecting Unix distribution, but not of Windows. So you'll need:

wget: http://gnuwin32.sourceforge.net/packages/wget.htm
At time of writing, the download page is http://sourceforge.net/projects/gnuwin32/files/wget/1.11.4-1/wget-1.11.4-1-setup.exe/download

gzip: http://gnuwin32.sourceforge.net/packages/gzip.htm
At time of writing, the download page is http://sourceforge.net/projects/gnuwin32/files/gzip/1.3.12-1/gzip-1.3.12-1-setup.exe/download

tar: http://gnuwin32.sourceforge.net/packages/gtar.htm
At time of writing, the download page is http://sourceforge.net/projects/gnuwin32/files/tar/1.13-1/tar-1.13-1-bin.exe/download

Install everything in the same directory and put the bin subdirectory in your PATH. hoogle data should run fine.

Note also that since it's using wget, if you have an HTTP proxy you need to configure wget to use it (Cabal uses the internet explorer connection settings and finds the proxy settings from that). The files is etc\wgetrc in the directory you installed wget, find the http_proxy entry and adapt to your settings.

EclipseFP: the way forward??

2011-06-24T11:50:00.001+02:00

This is not the usual "hey I managed to get some code to work, check it out" blog post. Bear with me as I'm just thinking out loud. This is about EclipseFP and how we can move it forwards.

At the moment EclipseFP works pretty well for simple, smallish Haskell projects. People have noted some performance issues on large projects, and part of it is the huge memory footprint of GHC when used through its API. After a couple of hours of working on a yesod project (so we're talking quasi quotation, template haskell, etc), the scion server takes 1Gb of memory... Restarting brings it back to reasonable levels, which seems to point to memory leaks in the GHC API.
Maybe more critically, people are now asking for features that get increasingly harder to implement using Scion:
- static GHC flags like -threaded (Nominolo's working on a version of Scion that could handle these)
- C or HSC sources (or other kind of stuff requiring a preprocessor)
- Having executables or test suites referencing the library component directly in the Cabal file

The main thing with these is that Cabal handles them fine, so basically typing cabal build at the command prompt solves these issues, which undermines the usefulness of having an IDE in the first place... But they require more than just calling the GHC API: building and linking a library in-place, calling a c compiler, hsc2hs, etc...
Of course Cabal publishes an API. So for example the code used by Cabal to build an in-place version of the library I could access, in theory, from Scion. But then Cabal calls the ghc executable, so the Cabal API gives me no easy way to get back errors from GHC, etc...

What does accessing the GHC API gives us? An AST, that can be generated from an unsaved file (so you don't need to save your haskell source file to see your errors, our up to date outline, etc). All the rest (syntax highlighting, jump to definition, etc) could probably be done from the AST.

So I'm wondering now if scrapping scion totally would not be a good idea. A rough outline of how things could work would be:
- create a temp folder in which we will work, copy in it all the project files (sources, etc)
- use cabal to build in that folder
- parse cabal output to get errors and their location
- unsaved files in the EclipseFP editor could have their unsaved contents copied into that folder (that folder represent the current state of the project, which may not be what is really saved on disk)
- have only a simple Haskell executable that given a module file, loads it using all the built files in the temp folder, and return the AST in some easy to parse format. Firing and getting the result would ensure GHC cannot grab loads of memory and not release it
- the IDE code then gets the AST and does all it needs to it
- the AST could even be saved in that temp folder in some format, so that the Java code would only request it when the original file is more recent, which would allow easy parsing of dependent modules AST (for example to retrieve all the symbols exported, etc)

There are a lot of ugly bits in there, maybe, but that would solve my problems... I suppose all of that could be written in Haskell and we could keep a similar enough API to scion, so that the Java code wouldn't need change too much...

For the moment I'm not going to reach into things anyway, but if anybody has some feedback, I'll gladly take any opinion on board!

Haskell + FFI + Java + SWT: crazy, maybe, but working!

2011-05-22T20:17:00.000+02:00

A few years ago I noted that being able to mix Haskell code and a Java GUI might be a good idea, since Java UIs have matured enormously, and Haskell UI library are still struggling to gain wide adoption, a recent thread on cafe being proof. SWT bindings were mentioned, so I had a look at both the Java and the C SWT code, for windows. Two immediate things sprang to view:
1. The SWT C library is using JNI, duh. It relies on JNI for some operations so you can't just use the SWT.dll in your haskell code that easily
2. The SWT C bindings are very low level, and there is a wealth of Java code on top of that to provide a lot ow glue and boilerplate code for all kind of widgets, and it would be long and painful to rewrite it all in Haskell.

My Java to Haskell code converter only being a dream at the moment (one day...), the next reasonable (ahem) option was then obvious: start a few blown JVM from Haskell and use it to drive SWT classes.
I'm a Java expert, I can say, and my Haskell is slowly coming on, but the glue in between has to be C, so it wasn't a smooth ride, but I finally got it working! So what's happening is this:
- The Haskell applications starts as they all do, in a main :: IO() function
- Haskell invokes some C code that starts a new JVM, passing it a classpath. The C code acts then as a facade hiding some of the intricacies of accessing the JVM
- Haskell uses the foreign C functions to create ands manipulate the SWT Java objects
- I even got callbacks working, meaning you can register a Haskell listener on a SWT button, and hence have Haskell code calculate the results of that button click.

And I have proof! Here's a simple SWT Shell with one button:

When you click the button:

And the handler for the button click is written in Haskell (I have a MVar called count for the number of times the user clicked):

do
    modifyMVar count (\c->do
        let nc=c+1
        let s=if nc==1 then "once." else ((show nc)++ " times.")
        text3<-toJString ("Clicked "++s)
        voidMethod button "setText" "(Ljava/lang/String;)V" [JObj text3]
        return(nc,())
        )

You can see here the invocation of the setText method of the SWT Button, which for the moment requires knowing the exact JNI signature, and the translation from Haskell strings to Java strings.

Of course, there is still loads to do. I probably need to automatically generate bindings to the Java objects to simplify greatly their manipulation, wrap stateful operations in a monad, etc. But this may open a new way for Haskell GUI. Or it may just be a crazy idea that has already taught me a lot about FFI and JNI...

Pirates 0.2: fixes and a chat interface

2011-04-18T20:26:00.000+02:00

I released a little update to my Pirates Yesod/Canvas game. Nothing too dramatic, but the new version, still at http://pirates.dyndns-free.com boasts:

Fixes in the collision detection code
A chat interface, so you can talk to the other ship you're fighting with. Stock up on pirates insults!
Fixes in handling of games when a ship leaves the game before it's finished

As I said, nothing too fancy, but I find that polishing off bugs and adding unplanned features is often a good exercise, especially for personal projects where the temptation is great to just move to something else when it kinda works. The chat interface was interesting: the server side code was done in a few minutes, using the same event system as the rest, but I struggled longer with the HTML to get the right positions for everything, as usual... Now it's on the right hand side, outside of the canvas area, which makes the full game area much bigger than the original 800x600, but I don't think I get a lot of mobile users anyways (-;.

I could probably make the game more appealing long term by having a bigger area where more than 2 ships could battle, and have more of a RPG feel to it (maybe be able to log in and out so you keep your ship, be able to repair it in harbours, etc...). I'll see what I have time to do. Any suggestions welcome!

Pirates!! A multi-player browser game with Yesod and Canvas

2011-04-08T14:12:00.000+02:00

I can finally open to the public a little game I've made! You can now go to http://pirates.dyndns-free.com and play!
The game is simple: you're sailing a pirate ship, and you have to sink your opponent. You can use cannons, ram the other ship, board it with your blood-thirsty crew. Your speed depends of course on the direction and strength of the wind. You can play alone versus a very dumb AI (mainly to get the hang of the controls), against a friend in a private game, or against any dog on the internet.
The technology stack is as follows:

Hosting is on a free Amazon EC2 instance. Performance is probably not going to be great, but should be sufficient I hope for the traffic I'll be getting!
Server side is Yesod 0.7. Coming from the Java/JSP/Struts world it was such a pleasure to work in Haskell, with type safe templates. I have no web server, just warp, since there are very few static files (a few sprites and the tile image).
Client side uses Canvas with a sprinkling of JQuery.

As you can see, no database. Information about current games is purely in memory, and I don't keep any long term information about players. I suppose that allowing players to log in with their Google or Facebook id, and keep statistics on their battles would be good, but I'm not sure this little game will get enough traction to warrant it, really...
The AI also is not great shake, there are only two bots: The Sitting Duck turns to face the wind and stays there. The Drunken Sailor just goes randomly through all possible actions. But hey, it's a multi-player game!
I'm not too sure if I chose the best route for concurrency. I use MVars and modifyMVar.
In fact the more difficult bit was to design the control. I didn't want to a do a full 3D environment, and opted for a top view, but the velocity of the ships was essential, and so providing just direction keys as in more static games wouldn't have worked. Hopefully the way the control works, being disposed on a copy of the boat icon at the top of screen, that turns when you boat turns, will work well. The icons are probably a bit small, it's fine for a mouse but I don't think it could work on a touch screen, unless you have tiny fingers.

A little bit of personal info: inspiration from this game comes from the French magazine "Jeux et Stratégies", that my dad used to get when I was young, and that got me into games. We have still kept all the issues, and issue 22 from August 1983 describes a simple ship battle game, rules written by Michel Brassinne (French warning!). Thanks!

Still it was fun to do!! Hope you enjoy playing it a little!

Install GHC7 and Yesod on Amazon Linux

2011-04-01T16:03:00.001+02:00

I've been working lately on a multi-player web game, to actually get to know HTML Canvas (on the client) and Yesod (on the server) properly. I have a first fully functional version of the game, so now is the time to think about hosting. Michael initiated a discussion about it, and in the meantime I decided to investigate Amazon since they offer free instances now on EC2.
The free AMIs (images) come with a version of Linux specific to Amazon, but apparently binary compatible with CentOS, on which GHC and the Haskell Platform have been reported to successfully build.
These are the steps I followed to get everything working, hopefully they can be useful to somebody else. So as per the Amazon Getting Started guide, you connect to your instance with the ec2-user.
First we install through the package manager some useful packages:

sudo yum install make
sudo yum install gcc
sudo yum install zlib-devel
sudo yum install glut
sudo yum install glut-devel

I didn't find gmp, so I downloaded and installed in the default location:

curl -O ftp://ftp.gnu.org/gnu/gmp/gmp-4.3.2.tar.bz2
tar -xjvf gmp-4.3.2.tar.bz2
cd gmp-4.3.2
./configure
make
sudo make install

But afterwards I had issues building some Haskell packages because it didn't like the default install folder of /usr/local/lib, it preferred /usr/lib, so I did:
sudo cp /usr/local/lib/libgmp.* /usr/lib/
But I suppose ./configure --prefix=/usr/lib/ would be the preferred way to do that.

Libbsd is also required otherwise unix-compat fails:

curl -O http://libbsd.freedesktop.org/releases/libbsd-0.2.0.tar.gz
tar -xzvf libbsd-0.2.0.tar.gz
cd libbsd-0.2.0
sudo make install

Then for GHC, I just downloaded the generic binary and installed that. I decided to install Haskell under my home folder.

curl -O http://haskell.org/ghc/dist/7.0.2/ghc-7.0.2-i386-unknown-linux.tar.bz2
tar -xjvf ghc-7.0.2-i386-unknown-linux.tar.bz2
cd ghc-7.0.2
./configure --prefix=/home/ec2-user/haskell
make install

Make sure we know how to find it:
export PATH=$PATH:/home/ec2-user/haskell/bin

Then get the Haskell Platform source and build that, since there is no binary (yet) for our platform:

curl -O http://lambda.galois.com/hp-tmp/2011.2.0.0/haskell-platform-2011.2.0.0.tar.gz
tar -xzvf haskell-platform-2011.2.0.0.tar.gz
cd haskell-platform-2011.2.0.0
./configure --prefix=/home/ec2-user/haskell
make
make install

Once you're there, it's back to familiar ground:

cabal update
cabal install yesod

And there we are!! My deployment platform is ready!

EclipseFP 2.0.4 released: supports GHC 7!

2011-03-21T19:41:00.000+01:00

Hello, the demand was HUGE (well, everything is relative :-)) for a version of EclipseFP that could build and run on GHC7. Since the 2.0.3 release there is now a Haskell Platform release that contains GHC7, so we had no excuse... And there it is! Thanks have to go mainly to Alejandro Serrano, for getting Scion to compile with GHC7 and to Thomas Schilling and Matti Niemenmaa, who upgraded their libraries for GHC7.

So what we have is a scion backend that should build both with 6.12 and 7.0, and hopefully work the same on both. We now support test-suite sections in Cabal files, with limited support for running test suites (running just cabal test with no options).

On my To Do list are full support for test-suites (having full launch configurations for Cabal defined tests) and HLint integration. But I'd also like to work on my own project, so don't expect that a release every two weeks is going to be our normal cruising speed.

As usual, just install from the Eclipse update site: http://eclipsefp.sf.net/updates, and bug reports or feature requests should go to the SourceForge forum or the development mailing list.

Happy Haskell Hacking!

EclipseFP 2.0.3 released

2011-03-07T19:34:00.002+01:00

It's my pleasure to release EclipseFP version 2.0.3. EclipseFP is a set of Eclipse plugins for Haskell development. Scooter and I have tried to add a few features and hopefully to fix the bugs that were reported by users.

Main things in this version are code completion templates, which enables you to code even quicker!! We also enhanced the Cabal support (you can now set Cabal flags from the project properties, run Cabal install on the project) and simplified the installation (more settings are auto detected).

EclipseFP is alive and well (I use it for all my Haskell work) so please send us all your feedback, bug reports, feature requests!

Use the standard Eclipse Updates Site or download files. Enjoy!

Evolving a computer with TECS and Genprog

2011-01-21T15:10:00.002+01:00

I've started reading The Elements of Computer Systems since it was recommended on Reddit and it's a great read. After the section on boolean gates I decided to have a quick go at build the gates in Haskell. Just implement Nand with pattern matching, and implement all the other gates just in term of functions calls to previously defined gates. The book mentioned "think positive" about implementing Not, so I just T as the second parameter, did't think more about it, even though I wasn't sure having a constant value was OK.

Then I moved on to the other gates. It was interesting to think about boolean logic without Or, for example, after years of only reasoning in terms of And Or and Not. But at some stage I thought "why am I doing this by hand?". I remembered a thread about a new genetic programming library on an Haskell group. Surely generating the gates would be simple enough?
So I got genprog from hackage, and adapted the example for my needs. I started with a very simple system to determine the implementation of Not:

And of course, the result came quickly enough: not a=nand(a,a). No need for a Const at all! Evolution has beaten me! (OK, I'm only a not-so-intelligent designer, I suppose...)
Now, I'm going to see if this approach can take me all the way to the arithmetic unit...

EclipseFP 2.0.2 released

2010-12-01T17:00:00.002+01:00

I am pleased to announce another release of EclipseFP. Version 2.0.2 should make installation of Scion easier, improve stability and performance of the parsing and building operations, and provide a few new functionalities. You can now configure where the "Open Definition" action will look for definitions (these can be in Haskell source code or Haddock pages): by default it will look in the dependent projects, in your GHC install doc folder, and on Hackage, but you can add more locations in the preferences. You can also generate source distribution via a project export command (that calls cabal sdist).

The release notes are here. To install or upgrade, just point your Eclipse to http://eclipsefp.sf.net/updates. Be sure to configure both the GHC path and the Cabal path in the preferences.

Happy Haskell hacking!

A simple game in Haskell + SDL

2010-11-25T20:07:00.000+01:00

I wanted to play a bit with SDL, and it's always good to step away from actually working on EclipseFP to use it for a "real" project. So I wrote a very primitive game using SDL and SDL_TTF. It's a typing speed game (hence the, ahem, clever pun on the name: TypeClass): you see letters scrolling down the screen and you need to type them, and of course they go faster and faster. Maybe my children will learn to type with it. Maybe not.

I had SDL installed earlier, but of course SDL_TTF proved challenging on my windows machine, but I think it's was all my fault, I probably didn't read the instructions properly or something. Anyway, I only forgotten to set the LIBS variable. When I ran export LIBS=/usr/local/lib/libSDL_ttf.dll.a in my MSYS environment then configure worked fine!

Afterwards it was all plain sailing, really. Just remember to add enableUnicode True in your initialization code so you can get the actual Unicode character typed (I'm on a French keyboard, so typing M gives me the SDL key ;). And of course you need a font file, so I include a GNU FreeFont file with the package. It needs to be in the same folder as the executable.

For some reason, after seeing code that was blocking for SDL events in a loop, I started building the game with two threads, one blocking for events and one looping and drawing, and a MVar in the middle. It worked, but rewriting it in a single normal game loop: process event, update game state, draw made things a lot easier.

This brilliant piece of software that has already made its mark in video game history can be found on Hackage here!

As usual, all feedback is welcome. Possible enhancements are a high score screen, difficulty levels, choosing letters by frequency in English, using Markov chains so that the flow of letters look more natural... The list is infinite!

EclipseFP 2.0.0 released!

2010-09-30T22:03:00.000+02:00

Hello all,

I'm proud to announce version 2.0.0 of EclipseFP, the Eclipse plugin for Haskell coding. The change in the major version number shows that some big changes have been made, even if under the hood:
- the Antlr parsing has been removed
- the JFace based syntax highlighter has been removed
We now rely solely on Scion and the GHC API to provide lexer tokens! This fixes some long standing syntax highlighting bugs while giving us more options for coloring.
Also, we now communicate to the Scion library via standard streams instead of network calls, which should solve some headaches.

There are also loads of bug fixes and small enhancements peppered a bit everywhere, either things that people asked or things that I did for myself, since EclipseFP is (of course) my IDE of choice for my Haskell coding. The release notes can be found here.

Another good news is that I'm not alone in hacking at EclipseFP, scooter has helped a lot with Helios, Cabal 1.6, GHC 6.10 and Mac OS X support, and has a few new features coming up soon!

Just point your Eclipse to the EclipseFP update site, and happy Haskell hacking!!

Haskell Neural Network: plugging a space leak

2010-09-24T14:43:00.001+02:00

First, the good news: last week when I posted about my little digit recognition program I had made a mistake in my test code: when I give a rounded eight to my network, it does recognize it, even though it's been trained only with a square eight! Yoohoo!

Also, a few people pointed to online database of handwritten digits so that I could get more data to train my network. I dully downloaded and started working with some files. Soon I knew I had a problem. Not only was the learning slow, but the memory usage of the program showed exponential growth, like using 400MB after a few iterations. Ha ha, I thought, so it's me against lazy evaluation again!

Well, I was right. A couple of strictness annotation in the neuron output calculation function helped performance, nearly making the code twice as fast, but didn't help the memory consumption. Going mad with ! (BangPatterns is your friend) didn't help any more. WTF?

Then I started to read a bit more about seq and Weak Head Normal Form. So when I put a BangPattern in front of a tuple, it evaluates the tuple but not what's inside it. And, lo and behold, if at each learning iteration I traced the network to the standard output, memory usage stayed constant! So I needed to tell Haskell to fully evaluate the network each time, so as not to keep thunks full of arithmetic operations hanging round. A distant memory of a package called deepseq provided me the final help I needed. I didn't use the package itself because I wanted to understand really what's going on, so I just roll out a few lines of my own:

And by calling deepSeq on my network in each iteration, my program runs in constant space! And I looked at the flat line of memory usage in my task manager, and I was pleased. Now, I can try to improve the speed of the dang thing, but that's another set of problems...

Digit recognition with a neural network. First attempt!

2010-09-17T17:17:00.000+02:00

I bought myself "On Intelligence", and I thought as the first step on the long road to building an intelligent machine I would go back to neural networks. I have my own little library that seems to work (kind of), so I decided I was going to try to write a little number recognition program. I was inspired of course by ai-junkie, but it was a bit short on details...
So here is the problem: I have a 5x5 grid of LEDs that can be on or off, and I want to recognize the digit that they show.
The input will be a list of 25 numbers, either 0 (off) or 1 (on)
The output will be a list of 10 numbers, either 0 or 1. The first position of a 1 indicates the result (so if the first element is a 1, it's number 0, etc).
I arbitrarily decide to set the number of hidden neurons to 50, twice the input neurons. So my topology is 25-50-10.
I train the network with only one version of each digit. For example 3 would be:

*****
    *
 ****
    *
*****

Wrote a few lines of Haskell code to read the training sets (moving from the representation with *, spaces and new lines to the list of 1 and 0, so that 3 becomes: [1.0,1.0,1.0,1.0,1.0,0.0,0.0,0.0,0.0,1.0,0.0,1.0,1.0,1.0,1.0,0.0,0.0,0.0,0.0,1.0,1.0,1.0,1.0,1.0,1.0]).
Then I train the network. It learns after 200 iterations, and successfully recognizes the training sets (ok, so things are working as they should be).
Then the real test: can my network recognize small variants of the training set?
Lets's try with:

****
    *
 ***
    *
****

(a slightly rounded 3): the network answers 3!!
Works also for a slightly rounded 6. But then:

 ***
*   *
 ***
*   *
 ***

(rounded 8) gives me 6. Oh dear... I could probably add more cases in the training sets and hopefully resolve the ambiguities, but there are bigger issues (I know I know, I'm very candid, but it's one thing being told something in a book, and another to build something to actually show it to you). For example:

***
* *
***

Is a small zero that doesn't take the full 5 LED width. My network recognizes 0 when it fills the full square, not when it's smaller (tells me it's a 5).
And of course, 1 is a disaster. The training "1" is a vertical row of LEDs on the left. A vertical row of LEDs on the right gives me 9, usually (and I suppose, 9 has all the LEDs on the right on).
So not only is basic training not sufficient to detect simple variations, but also the fact that we deal with absolute positioning of leds mean than scaling and simple side translations are not supported. So I suppose the next step is to not work with absolute positions, but relative positions of "on" LEDs. Another day's work...

EclipseFP development: using the GHC Lexer

2010-09-06T20:04:00.000+02:00

The new version of EclipseFP is slowly coming along. Slowly, because of some holidays, because of a lot of bug fixes, and mainly because of some big changes.
There were a few bugs revolving around syntax highlighting. The original code was using some lexing rules from the JFace packages in Eclipse, but it didn't cover some common enough cases of Haskell syntax, and of course was fairly hopeless at more complex stuff (things like Template Haskell). So I could either trod on and fix the lexing rules for everything, or I could use a different approach. So I decided to use Scion again, and to add support for lexing. Now Scion exposes a method that calls the GHC Lexer on an arbitrary Haskell source. We use the result to color highlight properly the Haskell source code. Of course I have to handle things like CPP preprocessing directives and literal source files myself, but that was ok. The two main issues I had to tackle were performance, and advanced lexing options.
Performance was poor at start, and there are two reasons: the underlying Scion code deals with JSON Strings while the socket code uses ByteStrings, and, I think, the Eclipse console. For the first issue, I'm using now AttoJson, so that Json marshalling is done with ByteStrings and try to use ByteString as much as possible in the code. The Eclipse console is a different matter. I think when too much data is written to it it struggles to update the console control, blocks the UI thread and the whole UI becomes unresponsive. For the moment, I have disabled server output in the console for commands (since now getting the tokens from the source code is quite a big response). Since lexing is also done in the UI thread, we've probably lost a bit in that area, but I feel the advantages (less Haskell knowledge in the Java code, more reliance on the GHC code, which I suspect knows about Haskell pretty well) outweigh the costs.
One funny thing with the Lexer I found: for example when you have TemplateHaskell code in your source file, and you have the TemplateHaskell pragma, the Lexer doesn't automatically set the TemplateHaskell flag. So I decided that I was going to enable all the lexer flags, since performance didn't seem to suffer. Be liberal in what you parse...

Anyway, I'm not going to release the next version right now, a bit of testing is still needed, and there are a few little things that I want to fix. The testing is easy, in a way: I use the development version of EclipseFP to hack the Scion code, so by eating my own dog food I hope that I can release something of value to Haskell programmers.
So don't worry, EclipseFP is coming soon!

EclipseFP 1.111 released

2010-06-02T15:59:00.000+02:00

I have just released version 1.111 of EclipseFP. You can get it from sourceforge or via the Eclipse update site (http://eclipsefp.sourceforge.net/updates/).
This is mostly a bug-fixing release to ensure compatibility with GHC 6.12.1 (Haskell Platform 2010) and IPv6. It also bundles a version of the Scion source code that it builds under the covers to save people from downloading the source code from github and building it themselves.

There aren't too many feature enhancements because I didn't get any requests for it. I'm not too sure a lot of people are using EclipseFP and are interested in seeing it improved. I suppose now with a Cabalized gtk2hs Leksah becomes more attractive. Anyway, if you have requests for EclipseFP let me know!

Gtk2hs and Leksah on Windows, check!

2010-05-28T16:12:00.000+02:00

Things have moved since I posted about the difficulty of being a Haskell programmer on Windows. Now, Gtk2hs has been cabalized and put on Hackage. I went into my MSYS prompt, and it installed like a charm! I've downloaded the latest version of Leksah, and it installed and started properly! As I write this I've seen the initial Leksah screen, and it's now busy referencing my packages (I wonder if giving it the location where Cabal stores the packages it downloads was a good idea, it seem to be taking a while...).
So I think now all the things I was struggling to get working are now installed and doing fine on my Windows 7 machine! There is hope!

Haskell SDL on Windows, check! (hacking required)

2010-05-05T16:48:00.003+02:00

Things looked grim. Even though Timothy could get it to work, the SDL Haskell library wouldn't build for me. libSDL itself installed ok and the samples compiled and ran. But I couldn't compile the hsc files because of a linker error: gcc couldn't find SDLMain or whatever. In desperation, I posted to Haskell-cafe. I didn't get a single answer or hint. I was truly alone.

OK, it was time to either give up or take drastic action. And I never give up.

By then I had narrowed the problem. The order of arguments to gcc were wrong for linking, according to the gcc manual: the .o file should be referenced before the SDL libraries on the command line. It would work like that for other people, but for me, on my system, in my corner of the universe, the order was wrong, and that was it. If I tried the same order of arguments on the SDL samples, they would fail to link. If I reverted the arguments, they would succeed.

So I was left with the straight and narrow: I managed to get the source code for hsc2hs from darcs. I fiddled for a while with the include-dirs and extra-lib options so I could get it to compile and produce the same error than I had with the Haskell Platform version, and not some stupid errors about hsFFI.h not found or so.

Then I opened up, hands trembling and sweat pouring from my forehead, Main.hs in an editor. Found the offending lines:

    rawSystemL ("linking " ++ oProgName) beVerbose linker
       (  [f | LinkFlag f <- flags]         
       ++ [oProgName]
       ++ ["-o", progName]
       )

Changed them to:

      rawSystemL ("linking " ++ oProgName) beVerbose linker
       ( [oProgName]
       ++  [f | LinkFlag f <- flags]
       ++ ["-o", progName]  
)

So that the name of the o file would be before the libraries.

Recompiled, copied the exe in the Haskell Platform bin folder...

Relaunched the SDL build...

And watched SDL build and install properly!

Gaped at the first sample of Timothy opening up a blank window that would close when I press a key! (OK, the last print fails, but by now little errors are not going to disturb my feeling of invulnerability).

Danced around my desk!

Had a beer!

Happstack on Window, check! (in a long, round-about, fear-inducing way)

2010-04-30T14:53:00.005+02:00

The other day I noted that I couldn't install the darcs version of Happstack on my windows machine. There was a build failure that was reported on the mailing list. So I waited a few days, and tried again. But then, before I started building Happstack, something happened. I don't exactly remember what, I think I was a bit to eager with cabal upgrade, and I upgraded some base packages that I shouldn't have. I remember directory-1.0.1.1 failing to build and having to go through MSYS to get it to build. Then more and more things needed to be upgraded and ghc-pkg check was reporting more and more weird things. When I thought I had upgraded everything, I tried happstack. happstack-util built, but then happstack-data reported that happstack-util couldn't be built because it required both time-1.1.4 and time-2-something. Oh oh. I screwed around for a while. Then I saw some posts on the web talking about removing package.conf.d. In my random experiments with cabal upgrade I had managed to install everything in my user profile and not inside my Haskell Platform directory, so there was hope!

I deleted package.conf.d. ghc-pkg list reported a saner list of packages. I tried building happstack again. Then something "funny" happened: every cabal build I was trying to do first tried to reinstall process-1.0.1.2, and failed every time!! And of course process-1.0.1.2 appeared in the ouput of ghc-pkg list... WTF? Ah, ok, directory-1.0.1.1 is still installed, let's kill the beast! And then everything worked! Happstack built, happstack "hello world" server even runs, life is happy again!

I didn't try to investigate why Cabal or directory-1.0.1.1 were insisting on reinstalling something that I have already. I've seen enough weird things and spent enough time this week away from the nice world of pure lazy functional programming and in dirty low-level configuration and build issues to not care too much anymore...

WXWidgets and WxHaskell on Windows, check!

2010-04-28T20:48:00.004+02:00

Despite my earlier problems, I finally managed to install WXWidgets and WxHaskell on my Windows machine. Pfiu! I started again from scratch, and things went much better when I installed, as recommended, MinGW and MSYS in different folders. When I need to compile stuff, I only put MinGW in my path, and when I need Unixy tools like make I add MSYS. Of course MinGW has its own version of make, mingw32-make, that works when MSYS make doesn't...

So I installed MinGW in c:\dev\MinGW and:

set PATH=c:\dev\MinGW\bin;c:\dev\MinGW\libexec\gcc\mingw32\3.4.5;%PATH%

Then I modified the WxWidgets config.gcc file to set SHARED=1 UNICODE=1 MONOLITHIC=1.

And then:

mingw32-make -f makefile.gcc BUILD=debug

worked!

Setting new environment variables (wxWidgets is installed in c:\ui):

set WXWIN=c:\ui\wxWidgets-2.8.10
set WXCFG=gcc_dll\mswud

Then I could install then wxdirect, wxcore and wx packages using a few more flags than what a default cabal install would do, for example:

runhaskell Setup configure --extra-lib-dirs=c:\dev\MinGW\lib --extra-include-dirs=c:\dev\MinGW\include --extra-include-dirs=c:\dev\MinGW\include\c++\3.4.5\mingw32 --extra-include-dirs=c:\ui\wxWidgets-2.8.10\include --extra-include-dirs=c:\dev\MinGW\include\c++\3.4.5

And it install, and works!

The sorry plight of a Windows programmer (aka me)

2010-04-22T17:03:00.003+02:00

Having secured fame and fortune with MazesOfMonad, my console based RPG, and having some ideas for some games that would look even better with a graphical front end, I decided it was time to take the bulls by the horn and go seriously into Haskell GUI programming. Well, the bull won and I feel truly gored...

So what's wrong? Well, after fooling around on my Windows 7 computer for a few nights I've achieved nothing. I now have three different copies of MinGW installed in different places. I have tried to installed SDL, GTK2HS, WXWidgets and Happstack and I have failed at all of them. I know WXWidgets is manageable because I've done it before, I only seem to have some include paths issues. libSDL itself installed but not the Haskell bindings. I found somebody with the same problem on the web, that then told me he had given up. I even managed to get Darcs errors with Gtk2Hs. In despair I thought I could develop a web based game but even the darcs version of happstack failed to compile, I remembered then some bug mentioned on the mailing list, so I should probably just download the latest stable version instead of the development branch.

Some problems are down to some silly things I did. For example installing the Haskell Platform in a directory with spaces is not a great idea (I don't remember if it was the default), because then a lot of paths referencing its MinGW install are wrong in the MSYS shell...

But most are due to the Unixy nature of a lot of the GUI components. Now I see a configure script and my blood freezes. I run "cabal install foo" and I get the dreaded message "this package has a configure script, you need to dive into Unix hell". Can you tell me why I have three versions of GCC, and two make a configure script crash with "c compiler cannot create executables" and the third one works? Where can I download one single file that will allow me to have a full MSYS/MinGW setup with most of what I need? I'm sure there are solutions to a lot of the problems I have, and maybe I need to spend more time reading the INSTALL files and such, and download everything from the MinGW files page. But why can't it Just Work? I mean, I work with Java at work, we do GUI work in SWT now and before that in Swing, we use a lot of libraries, and we never run into any installation problem. Maybe once or twice over the past couple of years we had a cross-platform compatibility issue or a library conflict. I know, I know, apples and oranges, but I'm now thinking that writing my own lazy, purely functional language with monads on top of the JVM with Java integration is going to be easier than getting a GUI library with Haskell bindings to work.

I like to write applications, not spend my free time trying to get libraries to install.

Found my neural network bug (three years later)

2010-04-09T17:26:00.004+02:00

After the university of Waterloo AI Challenge I went back to a bit of AI programming in Haskell. Three years ago I had written some code to implement a neural network, and it didn't seem to learn as fast as the book said it should. I had not worked on that any more, but I decided to look at it again. And I found a bug!! I mixed up the deltas to pass on from one run to the next. Changing two lines of code makes my network learning much faster.

So now onto something a bit more meaty than exclusive or: very simple character recognition. I want to be able to recognize numbers written as a digital alarm clock would display them (at least mine): you have 7 lights, four vertical and three horizontal, and various patterns make the numbers. If all lights are on, it's 8, if only the middle horizontal light is off we have zero, etc...

I just test things with a network that has 7 input neurons (one for each light), 10 output neurons (one for each number) and 8 hidden neurons (the average between input and output)

g<-getStdGen

n<-network 7 8 10

let trainingSet=[

([1,1,1,-1,1,1,1],[1,0,0,0,0,0,0,0,0,0]), -- 0

([-1,-1,1,-1,-1,1,-1],[0,1,0,0,0,0,0,0,0,0]), -- 1

([1,-1,1,1,1,-1,1],[0,0,1,0,0,0,0,0,0,0]), -- 2

([1,-1,1,1,-1,1,1],[0,0,0,1,0,0,0,0,0,0]), -- 3

([-1,1,1,1,-1,1,-1],[0,0,0,0,1,0,0,0,0,0]), -- 4

([1,1,-1,1,-1,1,1],[0,0,0,0,0,1,0,0,0,0]), -- 5

([1,1,-1,1,1,1,1],[0,0,0,0,0,0,1,0,0,0]), -- 6

([1,1,1,-1,-1,1,-1],[0,0,0,0,0,0,0,1,0,0]), -- 7

([1,1,1,1,1,1,1],[0,0,0,0,0,0,0,0,1,0]), -- 8

([1,1,1,1,-1,1,1],[0,0,0,0,0,0,0,0,0,1]) -- 9

]

let (LNS(n',err,ds),e) = run (initialState n) defaultLearningRate trainingSet (1,1000)

And after something like 150 iterations I get a network that has a low error rate. A little helper method to massage the results:

runNumber n inputs=do

let

(_, output)=exec n inputs

normalized=map (\v->if (1-v)>0.5 then 0 else 1) output

return $ elemIndex 1 normalized

Et voila!

Main> runNumber nn [1,-1,1,1,-1,1,1]

Just 3

Now, things are starting to get interesting! Hopefully I won't wait another three years before trying maybe more complex character recognition.

Waterloo Contest and Human Intelligence

2010-02-11T13:20:00.006+01:00

I've entered the Waterloo university Google AI Challenge, hiding under a cunning nickname, since I saw that you could develop in Haskell, and a few others Haskellers are on it too! At the moment, I'm cruising at around the 110th position, which is not too bad given that I only worked a few hours at night on it (I have a day job, you know...) and that I implemented everything myself since the starter pack wasn't ready when I started working. I've played a bit with the standard strategies, implemented some flood fill, reuse some A* implementation I have.
All of this is interesting, of course, but there is no AI as such. The only intelligence in my code is the one I put in, and the strategies I devised. So I'm thinking of playing around with genetic programming and evolution techniques to try to get my code to find the right strategies at the right time. I have the feeling the trick will be to design something that is not too restricted and is truly able to evolve into some more advanced and complex than what I've designed myself. But hey, I'll learn, even if my program doesn't (-:

EclipseFP 1.110 released, with debugging support

2010-02-01T16:20:00.002+01:00

Hello,
I have released a new version of EclipseFP. OK, I haven't kept to the one-release-a-month schedule, but with the Christmas season and everything, I deserved a break.
I have fixed a few issues here and there (thanks for everybody who commented on the blog or used the Sourceforge tracker or forum), and tried to externalize strings into resources, but there's still a lot to be done on that subject.
I have mainly worked at integrating the GHCi debugger with the Eclipse debugging framework. This tutorial was really useful! Now what works is that you can set breakpoints in the source, and GHCi will stop at them if launched in debug mode in Eclipse. Once stopped (either via a breakpoint or if you use :step) you can see the bindings and force their evaluation (use the set value to variable function in Eclipse). You can also create some expressions and get them evaluated. Underneath with use GHCi commands like :show bindings, etc.
I have removed some obsolete preferences and added debugging preferences (stop on error/exception, use show instances...).
I'll be happy with any feedback on all these features. One thing missing is support for :hist when invoking :trace, tell me if you'd like that.
Enjoy!

EclipseFP 1.109.0 is out!

2009-11-23T17:41:00.003+01:00

A month exactly after the previous release, I got another release of EclipseFP out! I tried to fix some of the issues that people reported, to improve the overall stability of the plugins and incorporated some new features that I hope could be useful (generation of extensions and ghc-options field in Cabal from the preferences, some quick fixes on GHC warnings...). I'll need feedback from users to know what features are really needed and could make a different to Haskell hackers.

On a related note, I read that in "Coders at Work" Simon Peyton-Jones talks about the need for Visual Studio and Eclipse based IDEs to get more people interested in functional languages. I hope that my modest contribution helps...

Releasing my code on the unsuspecting public (EclipseFP)

2009-10-23T15:41:00.004+02:00

Today I've released a new version of EclipseFP (Haskell support in Eclipse). There are quite a few changes in it, and there are even some changes in the Scion library that I've made and that Nominolo integrated. So people can give it a try if they want: get the current Scion source code, build/install it, and get the 1.108 version of EclipseFP.
Hopefully that's another step for me in getting more proficient in Haskell: interacting with other Haskell hackers, learning from them and their code, and releasing code that may actually been of use to somebody (or annoy them no end if it doesn't work) instead of only a little game as before. And of course working on an Haskell IDE, even in Java, gets me to learn new things about the language, the options for Cabal or GHC, etc...
Anyway, I hope I'll get some feedback on the new EclipseFP, and that we can move that thing forward!

What client for an Haskell Multi Player Game?

2009-10-09T12:59:00.002+02:00

I've been thinking again, in between working on eclipsefp, about writing a multi player role-playing game in Haskell. OK, I'm not going to write something the size of WoW by myself in my free time (unfortunately my day job doesn't involve any game programming nor any Haskell hacking :-(), but since I enjoyed writing Mazes of Monad, I figured that a game is a good way to learn things. A MMORPG should get me into concurrency, persistence, graphics, performance etc...

What I'm trying to figure out at the moment is what architecture to use. Mainly, if I have no doubt there is going to be a server somewhere running Haskell code, I wonder about the client. Should I try to write a browser based game coupled with a Haskell web server? That way, users could easily play the game when (if) it gets to a usable state, and I won't have to worry about cross platform issues for the client. Getting proper server side events or push should be fun, and of course if I end up using something like web sockets I will probably require a fairly cutting edge browser, which negates the advantage. I'm not too sure meddling with AJAX long polling and dynamic HTML is such a attractive option.
The other path is to write a Haskell client. Granted, users will have to download it, which means either me dealing with cross platform issues (provide several exes) or restricting my users to people happy to use cabal install. But I'll get maybe into stuff like OpenGL in Haskell, and do everything in Haskell instead of mixing Haskell with HTML and JavaScript (I've just stumbled upon jmacro, which could be cool I guess).

I'm sure everybody has faced that question at some stage. Ah well, for the moment I'm still wondering what my game will do, so I guess I have time to play with a few things before I make my mind up.

Haskell development in Eclipse

2009-09-28T08:53:00.002+02:00

I have started contributing to the eclipsefp project. For those who don't know, it's a Eclipse-based environment for developing in Haskell.
EclipseFP has had a troubled history so far. Leif started the project and developed an impressive set of features. He then pursued the Cohatoe project to be able to write Eclipse plugins inm Haskell, in a effort to avoid rewriting in Java all kind of things that already existed in Haskell. And then this summer Thomas started integrating with Scion, which is a Haskell library that communicates with JSON data over simple sockets with non Haskell clients. Scion offers method to load Cabal projects, type check files, etc... That work is not finished, but Leif and Thomas have done such a great job that it's easy enough now to finish the missing bits here and there.
This is perfect for me: part of the work is in Java and Eclipse, which is a lot of what I do in my day job, so I can bring a lot of experience there. Scion is a small enough library that need to be extended to provide me with all I want, so I can do little enhancements in Haskell and slowly build on features. And, last but not least, I need to learn the GHC API (which is what Scion uses to parse, check and compile files), to find what I need for.
This will only enhance my knowledge of Haskell, and I hope that I'll be able to make something useful to provide Haskell hackers with another IDE to develop in.
For those interested, the github repositories where my current work can be found are:

git://github.com/JPMoresmau/scion.git
git://github.com/JPMoresmau/eclipsefp.git

Yoohoo: mazes of monad is kind of popular!

2009-08-07T11:40:00.002+02:00

Don Stewart has published some statistics on packages popularity here . MazesOfMonad is the fastest rising application over the past 4 months. Thanks to all people who downloaded it, hope you enjoyed it, feel free to send me bug reports or enhancement requests. Hopefully I'll find the free time (in between day work, family, sports, etc.) to develop another more ambitious game in the coming months!

Mazes of Monad: 1.0.5 and (maybe) final version uploaded

2009-07-10T14:31:00.003+02:00

I've uploaded yesterday on Hackage the 1.0.5 version of my Role Playing Game called Mazes of Monad. I think this will be the last upload unless some big bugs are found. But I think I'll stop here in terms of functionality. It was fun, I learned a lot about Haskell, both on programming with the language, and on how to design Haskell code. Working on a full project over a long period of time forced me to actually understand code I had written previously and make it evolve instead of just having a clean slate.

I also realized than while developing a game is a lot of fun, the actual programming is the, ahem, easy part. I find that the actual game design (how to get usable and useful features) and the tweaking of the difficulty, for example, are the most challenging. For a while my test character was quite a strong warrior and I thought the game was easy, so I changed some settings to make it harder, then I realized that I had made it quite hard to start with a poor naked wizard that knew only one spell... I can see that balancing all the different game settings and getting the right formulas is critical, and that only comes from testing and collecting feedback...

Anyway, I have ideas for other games, probably the next one will be web based to have some kind of GUI and be multi player, maybe using happstack as the server (or should I just use a little home grown server in Haskell, for fun?). I'll keep on posting about that!

Adding a Writer Monad transformer

2009-05-15T16:00:00.002+02:00

I'm still working on MazesOfMonad, my Haskell RPG, now on code.haskell.org. It's actually fun to now have a working program and think about extensions in functionality, and then how to implement them in Haskell, instead of just playing around with the language.
I wanted to give the user more feedback that they currently got about what was going on in the game, and at the same time I wasn't happy with what I had written previously: I had the output messages as part of the method signature:

myFunction :: Type1 -> Type2 -> (Type3,[String])

So I decided a Monad was called for. The Writer Monad, in this case, which lets us append stuff together to get it all back at the end. I already had most of my game running in a monad transformer, so it was **just** a matter of figuring out how to wrap everything in a WriterT. I had a bit of fun at first, then got it working. So now my game runs in a

WriterT a (RandT (StateT b IO)) c

Where a b a are various types for the things I need: a is the type of messages I write, b the state I keep, and c the return value. RandT gives me random numbers, StateT gives me a state, IO allows me to do IO operations when I need to save the game or something like that.

The amazing thing, well, for me, is that a lot of my functions do not need to be aware of the whole transformer stack, but of only the type classes that they care about. My final transformer is an instance of both MonadWriter and MonadRandom. So my method that needs to write messages becomes:

myFunction :: (MonadWriter [String] m)=>Type1 -> Type2 -> m Type3

And I can write my code calling the MonadWriter methods to add messages. It's in a monad, but it's still purely functional: I return a Type3 object and write Strings. If my function needs random numbers:

myFunction :: (MonadRandom m, MonadWriter [String] m)=>Type1 -> Type2 -> m Type3

and I get random numbers capabilities!

The final code is now much cleaner than before (I actually don't use [String] directly for the MonadWriter type but an alias for it, so I can replace it with Seq String or something faster the day I fancy it). I think I now understand monads and monad type classes better, and I understand better how using a monad doesn't automatically mean using the IO Monad and lose purity. Haskell Nirvana is not far off, says he naively...

Haskell RPG Game uploaded on Hackage!

2009-04-27T16:40:00.003+02:00

I've worked on it on and off for months, I've rewritten parts of it entirely, it gave me the opportunity to knock together a real, complete application in Haskell, and it's finally done! I've uploaded my console based RPG game (read: nethack clone) to Hackage (get it here). You create a character and walk through mazes, trading, stealing, casting spells and fighting monsters.
You can look at the source code (no documentation, sorry) and see what I've used. I've used a monad to abstract away the state and the random generator (so I can plug a non random generator for testing, one that for example will guarantee that the next throw of the D20 die will give 8). I have developped my own little framework for console handling (executing actions from what the user typed in and displaying results) maybe I should look at the existing libraries and try to plug one in instead.
What I found was the most awkward was the record syntax for nested structures, I'll have to investigate to find an easier way to read and update my structures.
There are a few HUnit tests to verify that the most basic functionality works, so hopefully there is no huge bug.
It was fun to play around with Haskell and with game concepts. I hope I'll have time to work on even cooler stuff like adding AI for the monsters (so they can decide when to run away, etc...).
Any feedback on the game mechanism or on the code itself is welcome!

A high-level GUI library for Haskell

2009-02-27T15:28:00.003+01:00

There seems to still be a void in the Haskell world regarding GUI libraries. On one hand we have low level libraries based on well know solutions but doing everything in the IO monad, to cut short, on another functional reactive libraries that just don't seem to be accessible for haskell programmers of my level, somehow... And I'm not the only one: just the other day I saw a new attempt being made at a functional easy to use GUI library: Barrie. It looks promising, but I had already started my own, so I think I'm going to continue a little bit, I find I'm learning a lot in the process...

The ideas being my own library, tentatively called UITree are the following:

- you build your GUI using combinators that start from simple widgets and use composite widgets and layouts to augment a tree
- event handler can be attached to widgets in the tree. The handlers are of type (ui, state)->IO(ui,state): you get the current ui tree and the current state, and you transform them. The IO bit is to allow you to do IO actions if you want but you can do without.

- inside the event handlers you manipulate the ui tree a bit like JQuery does with HTML DOM: you can select particular widgets, operate on them, etc... Most operations are just transformers from the UITree type to the same type, so you can chain them.

- since you can't reference directly the widgets in your fonctional tree, you can give them explicit ids, so there is a disconnect between the actual tree and the handlers, which is similar to JQuery (you cannot rely on the compiler to make sure you manipulate an existing widget).

For the moment I got some small samples to work with WXWidgets (Barrie uses GTK, by the way).
The API is bound to change, but this is how a frame containing a close button and a button that increases a internal integer state and displays the current state:


simpleState :: IO (Int)
simpleState=do
 start simpleUI 0 -- start the WXWidgets UI with the given UI tree and initial state
   where
     simpleUI=(frame [text := "Simple"])  --  the enclosing frame
        (
         (panel [])  -- a panel to enclose our children
            (row [space := 5] -- children in a row, a bit of space
             [
               (button [text := "0"] [command :> (return . doInc)]),  -- a button showing 0 and increasing the state when clicked
               (button [wid:="close", text := "Close!"] [command :> (pureUI doClose)]) -- a button to close the frame, pureUI is just a helper function that shows we're not modifying the state
             ]
             )
           )
     doInc (tree,st)=(set [text := (show (st+1))] tree,(st+1)) -- set the button text and state to state + 1
     doClose tree=close $ top tree -- close the top of the tree (the frame)

Note how the ui building and the handlers are pure functions, so you just manipulate your tree and your state.

Predictable random for testing

2008-11-28T14:07:00.001+01:00

In my Haskell RPG game, the results of actions are random: I use the standard random generator to simulate the throw of a dice in a table top RPG.
However for writing unit tests that is not ideal, since I would like to be able to test the outcome of an action for a given result of the dice. I didn't
want to clutter the code (even if using a State monad) with something used only for testing, so I experimented with IORef and a custom Test Generator.

First of all, we need a data type to store our non random random generator. It will store a list of results and the index it's at in that list. It's
basically a circular list, so that when we reach the end we start again. If the list is one element long, then the same result will be given for all
dice throws:

data TestGen = TestGen [Int] Int

mkTestGen is just a helper function to initialise the index to 0

mkTestGen l=TestGen l 0

This is the real magic: I thought you needed to pass around an IORef as any other variable to be able to read its results, but in fact this is not necessary,
if somewhat of a kludge:

testGen :: IORef (TestGen)
{-# NOINLINE testGen #-}
testGen=unsafePerformIO (newIORef (mkTestGen []))

This means that testGen is initialized only once as a TestGen with an empty list, no matter how many time it's called. It seemed to work in ghci even
without the NOINLINE pragma. Don't ask me why. So, given a simple setter function:


setTestGen l=do
 writeIORef testGen (mkTestGen l)

We can specify what we want as the random generator. If we have an empty list, we use the standard random generator, otherwise we return the current element
in the list, and increment the index, going back to zero if we overrun the list:


randomInt ::(Int,Int) -> IO (Int) 
randomInt (l,h)=do
 (TestGen list ix)<-readIORef testGen 
 if null list
  then getStdRandom (randomR (l,h))
  else do
   let e=list!!ix
   let ix'=ix+1
   let ix''=if ix'==(length list)
    then 0
    else ix'
   writeIORef testGen (TestGen list ix'')
   if e<l || e>h
    then error (printf "%d not in range (%d,%d)" e l h)
    else return e

And this works, amazingly:


 setTestGen [2]
 l<-replicateM 3 (randomInt (1,6))
 assertEqual ("l is not 3 2s") [2,2,2] l
 setTestGen [2,3,4]
 l<-replicateM 4 (randomInt (1,6))
 assertEqual ("l is not 2,3,4,2") [2,3,4,2] l
 setTestGen []

(Notice how I reset the test generator to standard at the end by passing an empty list).

So now I can test my code with predictable results, that will be still random in production use.
Oops, already found a bug on the first test written with that technique!

Haskell for counting votes!

2008-11-03T22:06:00.005+01:00

There's apparently an important election going on in the States. There are also a few interesting articles, for example in the french maths magazine Tangente (no up to date online edition that I could find unfortunately) and in the New Scientist, on how the way you tally votes affect the outcome. Tangente has a stiking example, and I'll go through the 5 vote couting methods they outline with an Haskell implementation of each.
No pesky elephant vs donkey, here, the election is about your favorite female movie star:

data Star = MarylinMonroe | AngelinaJolie | BrigitteBardot | EmmanuelleBeart | ClaudiaCardinale
 deriving (Show,Read,Eq,Bounded,Enum,Ord,Ix)

Note the "deriving Ix" (from Data.Array.IArray) so we can use the stars as array indices.

We have six elector profiles: each profile has ranked the stars in order of preferences, and we have the numbers (in millions, say) of electors for that profile. For example, 7.2 million people prefer MarylinMonroe, then Emanuelle Beart, then Claudia Cardinale, 4.8 millions prefer Angelina Jolie, etc. Here's the full data:

votes :: [(Double,[Star])]
votes = [
 (7.2,[MarylinMonroe,EmmanuelleBeart,ClaudiaCardinale,BrigitteBardot,AngelinaJolie]),
 (4.8,[AngelinaJolie,ClaudiaCardinale,EmmanuelleBeart,BrigitteBardot,MarylinMonroe]),
 (4,[BrigitteBardot,AngelinaJolie,ClaudiaCardinale,EmmanuelleBeart,MarylinMonroe]),
 (3.6,[EmmanuelleBeart,BrigitteBardot,ClaudiaCardinale,AngelinaJolie,MarylinMonroe]),
 (1.6,[ClaudiaCardinale,AngelinaJolie,EmmanuelleBeart,BrigitteBardot,MarylinMonroe]),
 (0.8,[ClaudiaCardinale,BrigitteBardot,EmmanuelleBeart,AngelinaJolie,MarylinMonroe])
 ]

With this we need a few helper method before we're ready to implement our voting algorithms.

We need a way to tally votes for stars, taking into account that several profiles may vote for the same star at the same round (Claudia Cardinale at the first round, say). We use an array with the addition as accumulation function, sorted my popularity

accumVotes ::  [(Star,Double)] ->  [(Star,Double)]
accumVotes v=let
 arr=(accumArray (+) 0 (MarylinMonroe,ClaudiaCardinale) v):: Array Star Double
 in sortBy (\a b -> compare (snd b) (snd a)) (assocs arr)

For some reason I needed to tell GHC about the array type, it couldn't infer it.
We need a way to get the votes for a given round (even if we'll only be using the first round this way)

roundResults :: Int -> [(Star,Double)]
roundResults r=let
 v=map (\(a,b)->(b !! r,a)) votes
 in accumVotes v

And a simple extraction function to get the winner's name from accumulated votes:

bestVote ::[(Star,Double)] -> Star
bestVote v= fst $ head $ (accumVotes v)

Now we're ready. The first algorith is a simple one round voting election:

oneRound :: Star
oneRound=bestVote (roundResults 0)

This makes Marylin Monroe the clear winner. But a lot of countries use a two round system: the most popular candidates after the first round make it through to the second round:

We first need a helper function that retrieves the first element from a first list found in a second list (is there a standard method somewhere?)

findFirst :: Eq a => [a] -> [a] -> a 
findFirst toFind (a:rest) =  if elem a toFind
 then a
 else findFirst toFind rest

Then it's straightforward: we get the first two candidates after the first round, and accumulate their results over all the profiles

twoRounds :: Star
twoRounds= let
 (fr1:fr2:_)=(roundResults 0)
 allScores=map (\(a,b)->(findFirst [fst fr1,fst fr2] b,a)) votes
 in bestVote allScores

Angelina Jolie for President! A third method we can use is to proceed by elimination: at each round, we drop the least popular candidate, until we get only one:

elimination :: Star
elimination=  fst $ elimination' (roundResults 0)
 where
  elimination' ::  [(Star,Double)] -> (Star,Double)
  elimination' (a:b:[])=a
  elimination' l= let
   firstRound= map fst $ init l
   in elimination' $ filter (\a->(snd a) >0.0 ) $ accumVotes $ map (\(a,b)->(findFirst firstRound b,a)) votes

This little recursivity yields Brigitte Bardot. Borda suggested another method: we assign a weight to each candidate depending on its ranking, that weight is used as a multiplier of the votes. First choice gets a total tally of the votes times 5, second choice 4, etc.

borda :: Star
borda = bestVote $ concat $ map (\(a,ss)-> map (\(s,mul)->(s,mul*a)) (zip ss [5,4 .. 1])) votes

And this time, Emanuelle Béart comes on top! Last method we will survey is from Condorcet: we look at each match bewteen two stars in isolation. A match is won if more votes put the first star in front of the second star. We first calculate all the possible matches than consider each won match as 1 vote, and accumulate as before:

condorcet :: Star
condorcet= let
 matches=[[x,y] | x <-allStars, y <- tail [x .. ClaudiaCardinale]]
 matchResults= map (\match->(bestVote $ map (\(a,b)->(findFirst match b,a)) votes,1)) matches
 in bestVote matchResults

And, behold, the winner through this method is... Claudia Cardinale! Five methods, five winners!
Of course these are really quick draft of voting methods, they should take into accound draws, etc... Do not use this code to elect the president of a real country!

Arrays to the rescue!

2008-09-02T20:54:00.002+02:00

Some time ago I blogged about Haskell Generics and how they simplified my code of a little RPG game I'm writing. My data types were designed with lists. For example a character has traits, and traits are a list of Characteristic objects, that have a name (Strength, Dexterity, etc.) and values. The Generics code helped me to deal with finding the right item in the list depending on the name, updating its value, etc. Records were no good because you can pass the generated function as an accessor but not as a mutator.

I have some unit tests that generate random characters, and I thought they were pretty slow, which I thought was due to the random number generator. Then I ran the tests with profiling on. Guess what? The Generics code amounted for a huge amount of the time spent. And I'd say the underlying list operation, with loads of list updates, were not great either.

So I got rid of the Generics code, and I decided to replace my lists by something else. In the Java world (my day time job) I rely heavily on Maps. I thought it was a bit overkill in this case, so I turned to the Haskell array package. And there was light!

You see, Arrays in Haskell are a different beast that arrays in other languages. You can use different types for the indices, not only integers. So since the data type representing the names of characteristics is a simple enumeration, I could use that as the array index. So the characters traits are an array with the names as indices:


data Characteristic = Strength | Dexterity | ...
  deriving (Show,Read,Eq,Enum,Bounded,Ord,Ix)

Ix is the type class that an array index type must implement


type CharacteristicRatings=Array Characteristic Rating

data Character = Character {
  ...
  ,traits::CharacteristicRatings
  ...
  }

And Rating follow the same pattern, since a rating has the current value for the characteristic, the normal value, etc.

Then the array operations make it easy enough to read and modify values: ! to get the value at the specified index, // to update. I get type safety and expressiveness (no need to even use toEnum and fromEnum), and performance...

Performance? With the Generics code and lists, my tests took 8 seconds. With arrays and no Generics, 0.26s. Nuff said.

instance Data Map where -- half done!

2008-07-11T13:31:00.001+02:00

While testing my JSON serializer/deserializer, I made the (unpleasant) discovery that some standard structures (most notable Data.Map.Map) do not implement all of Data.Generics.Data interface. They just call error on things like toConstr and gunfold. Which is not good, since I use these methods. Why they can't just pretend that a Map K v is for generics a map of (k,v) I don't understand (they implement gfoldl that way), but hey, I'm only a humble learner of Haskell...

So I spent a few hours trying to modify my JSON code to work around this issue, to no avail (well I can get the serialization all right, but the deserialization seems to be tricky). So I've got another approach to work: since the default for Data instances is not right when your objects contains Map structures, lets rewrite these... Using the source for Data.Map and the documentation for Data.Generics.Data, I managed to get it to work, but of course the price is loads (loads, loads) of boilerplate code.

So here goes: I have the simple object: (assume import qualified Data.Map as M)

data MapObjF=MapObjF (M.Map String Int) String
 deriving (Eq,Typeable,Show)

So MapObjF contains a Map String Integer and a String

And here's the Data instance definition (big breath):


instance Data MapObjF where
     gfoldl k z (MapObjF m s) = k (k (z (MapObjF . M.fromList)) (M.toList m)) s
     gunfold k z _ = k (k (z (MapObjF . M.fromList)))
     toConstr (MapObjF _ _) = con_MapObjF
     dataTypeOf _ = ty_MapObjF

con_MapObjF = mkConstr ty_MapObjF "MapObjF" [] Prefix
ty_MapObjF   = mkDataType "MyModule.MapObjF" [con_MapObjF]

I define the constructor, the datatype, and implement gfoldl and gunfold transforming the map into a list of tuples. Of course if you have several constructors, loads of fields it soon becomes unwieldy. Now, is there a Haskell macro system so I can easily generate all that boilerplate for all my data types? Noooo I don't want to leave Haskell for LISP...

Handling errors in JSON to Haskell deserialization

2008-07-08T13:17:00.002+02:00

My JSON deserializing code presented previously works, but when an error occurs, most likely because the JSON doesn't represent the objects you expect it to, we're a bit lost: we get a terse error message and our program aborts, because Data.Generics tend to call error when it encounters a problem, and our own calls from Maybe to Just may fail. So we need to build a way to report errors without crashing. Now, famously, there are several ways to report errors, and for a start I'll work with Either.

First, we create a error type that can help us represent both JSON parsing error and deserialization errors:

data JSOND13NError=JSONError ParseError
  | D13NError String String
  deriving Show

The two parameters in D13NError are the error message and the path in the JSON object where the error occured. They will be carried in the Either monad in a tuple of Strings

So parsing can either throw a JSONError or try to deserialize:

jsonStringToObj :: forall a. Data a => String -> Either JSOND13NError a
jsonStringToObj s= case parse JSON.json "JSON.parse" s of 
 Left err->Left (JSONError err)
 Right js->case (jsonToObj js "/") of
  Left (s,path) -> Left (D13NError s path)
  Right o-> Right o

The path starts at "/" for the deserialization

To work with Either we need a couple of helper function, one to translate the Maybe values we get from cast to Either:

toEither :: String -> Maybe a -> String -> Either (String,String) a
toEither s Nothing path=Left (s,path)
toEither _ (Just a) _=Right a

And then the monad declaration for Either(message,path):

instance Monad (Either (String,String)) where
 return a=Right a
 fail s=Left (s,"/")
 Right a >>= f2 =f2 a
 Left s >>= f2 =Left s

With this, our main function becomes:

jsonToObj :: forall a. Data a => JSON.Value -> String -> Either (String,String) a
jsonToObj (JSON.String s) path=toEither "Not a String" (cast s) path
jsonToObj (JSON.Bool b) path=toEither "Not a Bool" (cast b) path
jsonToObj x path=do
 (values,cons)<-case x of
   JSON.Object m -> fieldJSONValues m myDataType path
   JSON.Number fl -> Right ([],if isPrefixOf "Prelude.Int" (dataTypeName myDataType)
    then mkIntConstr myDataType (round fl)
    else mkFloatConstr myDataType fl)
   JSON.Array [] -> Right ([],indexConstr myDataType 1)
   JSON.Array (x:xs) -> Right ([(x,path),((JSON.Array xs),path)],indexConstr myDataType 2)
 let StateT f=fromConstrM (StateT (\((x,path):xs) -> do
   r<-jsonToObj x path
   return (r,xs)
   )) cons
 r<- f values
 return (fst r)
 where
  getArg :: a' -> a'
  getArg = undefined
  getType :: a
  getType = undefined
  myDataType = dataTypeOf (getArg getType)

We use a StateT monad transformer to wrap our Either monad in the State. I'm not sure I understand 100% of how monad transformers actually work, but it works!

And the fieldJSONValues handles a few error conditions, along with building the path where we're at, by adding after the current path the name of each field in turn

fieldJSONValues :: (M.Map String JSON.Value) -> DataType -> String -> Either (String,String) ([(JSON.Value,String)],Constr)
fieldJSONValues m dt path | isAlgType dt=
 if idx>(maxConstrIndex dt)
  then Left ((printf "Constructor index %d too big" idx),path)
  else 
   if null fn 
    then Right ((map (\x->(fromJust $ M.lookup (show x) m',path++(show x)++"/")) (sort $ map (\x->((read x)::Int)) (M.keys m'))),c)
    else  
     let vals=map (\(x,y)->(fromJust x,y)) (filter (\(x,y)->isJust x) (map (\x->((M.lookup x m'),path++x++"/")) fn))
     in if (length vals) < (length fn)
      then Left ("Not enough fields",path)
      else Right (vals,c)
 where 
  idx=case M.lookup constrIndexField m of
   Just (JSON.Number f)->truncate f
   _ -> 1
  m'=M.delete constrIndexField m
  c=indexConstr dt idx 
  fn=constrFields c
fieldJSONValues m dt path | otherwise =Left ("Not an algebraic type",path)

Once I had the previous code working, using a different monad and adding error handling was easy. I suppose I could further abstract and try not to hard code the Either monad in it but use any type of Monad or MonadError, but this gives me what I want: no program crash and the ability to recover from errors!

Deserializing JSON to Haskell Data objects

2008-06-27T11:25:00.002+02:00

Since last week I found a few other posts and libraries on the subject of JSON serialization and deserialization: here and here for example. Nonetheless I've continued on my own path, since the best way of learning is doing. It took me a while to figure out how to reconstruct a Haskell object, and I've only got some limited functionnality to work, but it works. I looked into the source code for gread for clues, I have to say.

So what we want is simple enough:

jsonToObj :: forall a. Data a => JSON.Value -> a

Given a JSON value we want an haskell object, and hopefully the type will match... OK, I'm a bit terse on the error handling side of things at the moment!
(Note: for the code to work you need to import: Control.Monad.State, Data.Generics, Data.List, qualified Data.Map as M, Data.Maybe)

It's simple enough for Strings and Bools:

jsonToObj (JSON.String s)=fromJust $ cast s
jsonToObj (JSON.Bool b)=fromJust $ cast b

Of course, if you expected a Foo and parse a JSON.String, this will fail(The cast return a Maybe). You have to cast since the signature doesn't force Strings or Bools, only Data.

For other types of objects (including numbers, because cast doesn't perform number conversion I found) it's a bit more complicated. Basically we have to figure out the type we want, find a constructor to build an instance of that type, and pass to that constructors proper values from the JSON object.
To find the type we want, I use funny code find in gread:

myDataType = dataTypeOf (getArg getType)
getArg :: a' -> a'
getArg = undefined
getType :: a
getType = undefined

This create a dummy function returning my type, and a dummy function returning what it gets as parameter. They don't need to be implemented, the compiler only cares about the type.

The meat of the function, that decides what constructor to invoke and what data to useas constructor parameter is a bit more involved:

(values,cons)=case x of
 JSON.Object m -> let
  c=indexConstr myDataType 1
  in ((map (\x->fromJust $ M.lookup x m) (constrFields c)),c)
 JSON.Number fl -> ([],if isPrefixOf "Prelude.Int" (dataTypeName myDataType)
  then mkIntConstr myDataType (round fl)
  else mkFloatConstr myDataType fl)
 JSON.Array [] -> ([],indexConstr myDataType 1)
 JSON.Array (x:xs) -> ([x,(JSON.Array xs)],indexConstr myDataType 2)

This snippet calculates the objects to iterator over and the constructor index to use. There's a bit a hand waving there: for objects we will take the first construtor we defined (I'll implement multiple constructor support later), and the JSON values it contains in the map. We just ensure we get the values in the order the fields are defined (given by the constrFields function). For number we use the int constructor if our result type looks like an int, otherwise we use the float constructor. There is probably a better way to construct a number from a Double, but I still need to find it. For the moment we look if the type name starts with "Prelude.Int", which is arguably not very Haskelly. For Arrays, we need to recreate the (head,rest) tuple that gave me trouble when serializing, so we deal first with the head of the list, and put the rest afterwards. The empty list is the first constructor, a non empty the second.

Then to actually create the object we pass the values inside a State monad:

State f=(fromConstrM (State (\(x:xs) -> (jsonToObj x,xs))) cons)

For the list of values we convert the head from json and keep the rest as state. This simple line was the result of intense thinking, I implemented my own State monad first to really understand what I needed to do and then figured out that the State monad did the exact same thing. The final working code is always shorter that all the previous failed attempts!
We then only need to call the State function with the values we calculated earlier, and that give us our full code:

jsonToObj x=
 fst $ f values
 where
  getArg :: a' -> a'
  getArg = undefined
  getType :: a
  getType = undefined
  myDataType = dataTypeOf (getArg getType)
  (values,cons)=case x of
   JSON.Object m -> let
    c=indexConstr myDataType 1
    in ((map (\x->fromJust $ M.lookup x m) (constrFields c)),c)
   JSON.Number fl -> ([],if isPrefixOf "Prelude.Int" (dataTypeName myDataType)
    then mkIntConstr myDataType (round fl)
    else mkFloatConstr myDataType fl)
   JSON.Array [] -> ([],indexConstr myDataType 1)
   JSON.Array (x:xs) -> ([x,(JSON.Array xs)],indexConstr myDataType 2)
  State f=(fromConstrM (State (\(x:xs) -> (jsonToObj x,xs))) cons)

Serializing Haskell objects to JSON

2008-06-20T14:21:00.002+02:00

I'm trying to do some simple serialization of Haskell objects to save some state to disk. After tearing some of my hair out debugging parse errors due to silly code I'd written in Read instances declarations, I've decided to use another approach. I'm going to save objects as JSON, since I've already used that JSON library.
The first task is to write generic code that can serialize a Data instance to JSON. This has probably been done somewhere else, but I need to learn, right? So I took a deep breath and dived into Data.Generics.
I quickly rounded up the issues I would face. Most notably, Strings and lists are accessed without any syntaxic sugar, so to speak: Strings are lists of Char, and lists are made of two fields, the head and the rest. Of course, you say, but from that I need to regenerate String and lists of JSON Value objects.
So, to start, how to recognize Strings to treat them differently than other algrebraic types:

isString :: Data a => a -> Bool
isString a = (typeOf a) == (typeOf "")

There's probably better ways to do that, just tell me (-:

Lists (that are not strings) can be recognized with abstract representations of constructors, which are equals regardless of the actual type of elements in the list

isList :: Data a => a -> Bool
isList a
   | isString a = False
   | otherwise  = (typeRepTyCon (typeOf a)) == (typeRepTyCon $ typeOf ([]::[Int]))

Now, transforming a list of the form (head,rest) to [head1,head2...]

jsonList :: Data a => a -> [JSON.Value]
jsonList l=
   concat $ gmapQ f l
   where f b
       | (isList b)= jsonList b
       | otherwise = [objToJson b]

For each element (the actual number depends on whether we're the empty list or not) we either reapply the same method, if it's the inner list or we simply transform to JSON

And then the actual method on objects:

objToJson :: Data a => a -> JSON.Value
objToJson o | isString o=JSON.String (fromJust $ ((cast o)::(Maybe String)))
objToJson o | isList o=JSON.Array (jsonList o)
objToJson o | otherwise=
   let
       c=toConstr o
   in
       case (constrRep c) of
           AlgConstr _-> JSON.Object (Data.Map.fromList(zip (constrFields c) (gmapQ objToJson o)))
           StringConstr s -> JSON.String s
           FloatConstr f -> JSON.Number f
           IntConstr i -> JSON.Number (fromIntegral i)

We first handle Strings, then list, then general objects using constrRep to distinguish between algebraic types that create JSON objects with the proper field names (using constrFields) and other types for JSON primitives.

And that's it for the serialization! The Generics package is not that hard to use but you have to look up examples to figure out how actually use the functions like gmapQ and such...

Now, I have to work on the opposite process: given a type and JSON data, reconstruct the objects... More Haskell fun!

The unintentional infinite list

2008-04-11T16:00:00.002+02:00

The other day I was happily hacking away at some Haskell code. Same thing as always: think hard at about how to represent my problem in Haskell, then type the code quickly enough. Then launch my HUnit test. Tests doesn't return, just seem to be looping for ever!! Uh oh.
So I check my code, all the foldr and foldl and such, to make sure nowhere I create an infinite sequence. Nothing. So I took a deep breath and started the ghci debugger. I'm used to graphical debugger (like Java in Eclipse) but hey I managed to pretty quickly locate the offending function. Only a typo that couldn't be caught by the compiler. Since it's not the first time this has happened to me, here it is:
I tend to not be too comfortable with big one liner functions, so I revert often to several lines after a let instruction to cleanly separate each step of the computation. So when I repeatedly change some structure I tend to get:


fn a =
   let
       a1=fn1 a
       a2=fn2 a1
   in fn3 a2

and the typo there was a line like:

    a2 = fn2 a2

So I had typed in a2 instead of a1, and the compiler was perfectly happy, I wanted to create an infinite list, right? Er... That wasn't exactly my intention.
So I fixed the typo. Then ran my test. Which passed. Of course.

Me and my compiler

2008-04-01T16:10:00.005+02:00

Pfiuu, long time no post, between my day time work, playing with Haskell and Lispy things (cool sutff like Clojure) and my extra-programming activities (yes, yes)...
I was just thinking about dynamically vs statically typed languages, since there is a lot of discussion about that on the web these days. For example, this article talks about refactoring in both static and dynamic languages. Well, I'm not a huge fan of all the refactoring utilities Eclipse give me, but there's one feature I use a lot: you know, when I change something, like the return type or the arguments of a method, my IDE helpfully adds little red crosses on all the source files that won't compile any more. So then I go through them and adapt the code.
Of course in a dynamically typed language I wouldn't get that, but since everybody has 100% code path coverage in their unit tests, it's just deferring the red cross marking a bit lower than the chain, right? Well, at work it takes 30 minutes to build our whole software from SVN (and only a few seconds to rebuild one individual module from inside Eclipse), and several hours to unit test everything (and have we got 100% test coverage?), so... Methinks I'll stick with something that can check that my types are coherent for me when I work on big projects...

Scraping my boilerplate: Generics instead of Records

2008-01-18T16:18:00.000+01:00

I was talking last week about my trouble writing simple code to access and update record field in Haskell. While talking with justin about it, he suggested using parameter type for record object, so that the compiler would check that an update method would actually take as parameters the proper rating and the corresponding update function. That was cool enough, and that got me into the "Scrap your boilerplate" papers and Data.Generics modules. What I have now, using generics programming, is fairly simple and I think I'm getting closer to what Haskell code should be like.
First, Ratings: a Rating has three int values: the current level, normal level, and the experience points. What I do now is that each value is actually typed, and the Rating only holds a list of them:

data RatingScoreType=Normal |
    Current |
    Experience
    deriving (Show,Read,Enum,Eq,Typeable,Data)

data RatingScore=RatingScore RatingScoreType Int
    deriving (Show,Read,Typeable,Data)
data Rating=Rating [RatingScore]
    deriving (Typeable,Data)

With that I can add an update method that only changes a RatingScore if the type match with the type provided:

addRS :: RatingScoreType -> Int -> RatingScore -> RatingScore
addRS  t2 b rs@(RatingScore t1 a)
    | t1==t2=RatingScore t1 (a+b)
    | otherwise=rs

And a method that tells me if a score match the given type

getRS :: RatingScoreType -> RatingScore -> Bool
getRS t2 rs@(RatingScore t1 a) 
    | t1==t2=True
    | otherwise=False

(note that addRS can be rewritten using getRS but we don't gain anything in code size)

I can then use Data.Generics functions to provide generic update and get functions:

addR :: Data a => RatingScoreType -> Int -> a -> a
addR rst i=everywhere (mkT (addRS rst i))

getR :: Data a => RatingScoreType -> a -> Int
getR rst a=
    let (RatingScore t1 b)= head $ listify (getRS rst) a
    in b

(There are probably other and maybe better way of writing these, but hey, the documentation is not very rich in examples...)

The level above Ratings are Characteristics: a character holds an array of CharacteristicRating:

data CharacteristicRating = CharacteristicRating Characteristic Rating
    deriving (Show,Read,Typeable,Data)

Where Characteristic is again an Enum of all possible characteristics.

I define the similar 4 functions as above, except they filter on Characteristic first and RatingScoreType second:

addC :: Characteristic -> RatingScoreType -> Int -> CharacteristicRating -> CharacteristicRating
addC c2 rst i cr@(CharacteristicRating c1 r) 
    | c1==c2=everywhere (mkT (addRS rst i)) cr
    | otherwise=cr

getC :: Characteristic -> RatingScoreType -> CharacteristicRating -> Bool
getC c2 rst cr@(CharacteristicRating c1 r) 
    | c1==c2=True
    | otherwise=False

addCharacteristic :: Data a => Characteristic -> RatingScoreType -> Int -> a -> a
addCharacteristic c rst i a = everywhere (mkT (addC c rst i)) a

getCharacteristic :: Data a => Characteristic -> RatingScoreType -> a -> Int
getCharacteristic c rst a = 
    let cr=head $ listify (getC c rst) a
    in getR rst cr

This is pretty cool: if I have a Character c:

getCharacteristic Strength Current c

Gives me the current strength! And a function can take a Data instance (a Character or anything else) and a Characteristic and do both testing and changing value in perfect safety!

Now I'm only just starting playing the Data.Generics, and more generally with Haskell program design, but this is cool. My RPG is not progressing fast, but the main game here is actually building it, not playing it!!

Nested records headaches in Haskell

2008-01-11T13:33:00.000+01:00

I'm fighting a bit with Haskell records and named fields. This post has some valid remarks, but no real solutions. The main problem I have is managing elegantly nested records:
In my (embryonic) Haskell RPG game, a character is defined as a data type with several named fields (name, gender, traits). Traits is a collection of all characteristics (Strength, Dexterity, etc.). Each trait is itself a record with three values (normal level, current level, experience points). When an action is performed, not only we look at a rating value, we also update the rating experience as a result. So I want a function that takes a character, the name of a characteristic, and can update the proper characteristic rating.
With records
Traits { strength::Rating...}
I get a lookup function: a function strength that I can use to get the current value, so I can have a function like
testTrait :: Character -> (Traits -> Rating) -> ...
Where ... is whatever result the function returns (something like IO(Character,TestResult). But if testTrait needs to modify the record, I don't have a dynamic function to do it! And even if my code could hard code that "strength" is the trait I need, then the update mechanism is singularly cumbersome, since:
let c1=c{traits{strength{experience=1}}}
Doesn't compile (c is a character, I want to set experience of strength to 1). And if I want to modify the value (like do a +1 instead of setting to 1), then I need to unstack everything and rebuild it. Argh! Surely there is a better way?

The solution I have for the moment is that Traits contains only a list of Rating objects. Then I've defined an Enum with a constructor for each characteristic:
data Characteristic = Strength | ...
deriving (Show,Read,Eq,Enum,Bounded)
and I can have a method that combines fromEnum and !! to retrieve the right value. I can also have a little function that let me update a precise element in a list. But of course the type system does not guarantee me that every Character will have the right amount of Rating objects in the array!

It may be that my brain has been totally formatted by years of Java (in Java I could just pass the Rating object and it would get modified in place of course), but I just don't know what structure would give me total safety (enforcing that every character has the proper data) with no boilerplate code and no duplication. (I know I can define helper methods to do all of that, but I though Haskell would let me cut the amount of purely technical code needed and concentrate on the business).

Haskell syntax is not Java syntax; good or bad?

2008-01-11T12:43:00.000+01:00

In this blog, the author expresses the opinion that an important part of a language popularity: Java was successful because it looked familiar to C++ developers, and similarly new languages should use Java syntax. This struck a chord: after all, I have started writing a pure functional language using Java syntax. I thought that having a similar syntax would help grasping the language better. But now I wonder. Not only do C++ and Java share a similar syntax, they also share some concepts (objects, etc...). If a language is intrinsically different (like a pure functional language with monads, say), then having a different syntax tells you straight away: warning! This is different, so not only you must learn a new syntax, but you must also learn a new way of programming.
So maybe having Haskell using a totally different syntax is A Good Thing. And it gives me a nice excuse to stop trying to create my own little language that looks a bit like Java but doesn't act like Java and concentrate on writing Haskell code.

Java and higher order generics

2007-12-21T08:37:00.000+01:00

Funny coincidence: yesterday in work I was trying to find a way to express something simple enough in Java: I want a method map that takes a generic collection and return a collection of the same type (that is, passing a List should return a List, passing a Set should return a Set) with a different generic type. In Java you can have:

public <T1,T2> Collection<T2> map(Collection<T1> c,Mapper<T1,T2>

Where Mapper is an interface defining the method T2 map(T1 o);
There is no way I think in Java to express that if you pass a subclass of collection you get the same subclass back.

I tried things like <T1,T2,C extends Collection,C1 extends C<T1>,C2 extends C<T2> and other crazy things that my compiler didn't appreciate.

You have to duplicate methods, and that's what we've done in work:

public <T1,T2> List<T2> map(List<T1> c,Mapper<T1,T2>
public <T1,T2> Set<T2> map(Set<T1> c,Mapper<T1,T2>

And of course a colleague needed a Vector because he was using an old library still using Vectors...
And then today I found that blog post, that point to that PDF document. Apparently you can do this kind of things in Scala and Haskell... Do I need to convince my company to move to Scala?

Haskell design patterns are (probably) needed

2007-11-30T14:19:00.000+01:00

I was reading that post, and thinking: well when I start a Haskell project (for fun, yes, I don't use Haskell in work) I do wonder how best to design it. Yes I know about higher order functions, generics and such, but I don't think these low level constructs are enough to express, on their own, the design of the system as the design patterns from the GoF do. I would love to see patterns on how to allow extensibility, separation of concerns and such in Haskell. Years of experience in Java mean that I see a lot of things in terms of objects, inheritance vs composition, interfaces, etc, and not all of that apply in Haskell. I see a lot of low level algorithms in Haskell, but few software design papers (papers like this one are useful, we need more of them).

Structure of a functional Java, er, method

2007-11-26T08:30:00.000+01:00

A method in FLOJ (remember, the functional Java based language I'm toying with) has a simple structure:
- a method definition 100% similar to a normal Java method: visibility generic return-type type parameters.
There are no exceptions, of course, since exceptions are things Best Done in Monads, once I've figured out
out I'll implement monads.
- a series of assignments of the form x=e where x is a variable name, and e an expression
- a return expression

For example:


public static String[] testMap(){
   a=["toto","titi"];
   a.map(String.toUpperCase);
}

Since the return expression is always the last expression, there is no need for a return keyword, but it's accepted by the parser. In the exemple above, the method returns the list of String TOTO and TITI.
Note also that the assignments do not allow nor require the type definition of the left hand side of the assignment. The type is inferred from the
return type of the right hand side expression (in the exemple above, a is a list of String). It's not as powerful as the full blown Haskell type inference but at least it saves some typing.

Of course, the generated Java code will add a sneaky final keyword to all the assignments, to enforce the fact that there's no variables in a functional language.

All in all, that structure makes the code easy to read and to understand: the sequence of calculations and the final expression yielding the method result.

I don't think my FLOJ language will ever be a full blown industrial-strength language, but I know that by implementing my own functional language I will learn a lot
more about functional languages than by just using one. Just implementing method pointers, type inference, combinator parsers and such, I understand more the why and how of Haskell.

A simple functional structure in Java

2007-10-26T12:21:00.000+02:00

I have progressed a little bit in creating a functional language on top of Java, code name FLOJ (Functional Language On Java (-:). For the moment I'm only working on single files (I have a working parser), on the simple language constructs.
It's really interesting, because looking at very simple things make you think in depth about some issues that you may not really consider in Java proper. For example, this is the definition of a very simple structure:

package fr.moresmau.jp.floj.samples;

public class Simple(String data,int port){

}

So you can see that the package system is Java's and class definition are very similar. There are no constructor, so. Since null are not allowed (nulls are a huge source of bugs, I'll provide a Maybe object or something like that later),
the structure must have all its data filled in. If you want different operations to be done, you just create static functions that take parameters and create what you need, no need for several constructors. So the list of fields is entered straight after the name of the class.
Simple is a structure with a string and an int. It's easy enough to generate the constructor in Java:

package fr.moresmau.jp.floj.samples;

public class Simple{
 public final String data;
 
 public final int port;
 
 public Simple(final String _data, final int _port){
  if(_data==null){throw new NullPointerException("_data");}
  this.data=_data;
  this.port=_port;
 }
...

The fields are immutable of course, so we don't bother with a getXXX() method, we just expose the field.
And you can then add setters that return a new object :


... public Simple setData(final String _data){
  if(_data==null){throw new NullPointerException("_data");}
  if (_data.equals(data)){return this;}
  return new fr.moresmau.jp.floj.samples.Simple(_data,port);
 }
 
 public Simple setPort(final int _port){
  if (_port==port){return this;}
  return new fr.moresmau.jp.floj.samples.Simple(data,_port);
 }
...

Note the null handling is not ideal, left for later. Note also we avoid superfluous creations if the data will not change.

So far, so good.

Then I decided to implement equals, so that equals is based on the data and not pointer equality:


...
 public boolean equals(final java.lang.Object o){
  if (this==o){return true;}
  if (o instanceof fr.moresmau.jp.floj.samples.Simple){
   fr.moresmau.jp.floj.samples.Simple co=(fr.moresmau.jp.floj.samples.Simple)o;
   if (!this.data.equals(co.data)){return false;}
   if (this.port!=co.port){return false;}
   return true;
  }
  return false;
 }
...

And with this of course the problem of overloaded equals method raises its head. If I want to allow subclasses to add more fields, my equals method won't work for subclasses.
If I check the exact class name in equals, subclasses that only add/overload methods will not be equals to superclasses.
So what can I do? I haven't decided. I could forbid subclasses to define more data fields (that's I think what you have in Haskell: you inherit functions from your type classes but no data).
I could also make the equals method final and not generate it for subclasses, but equality would then ignore the additional fields.
Another option would be to check the class name in that equals method and generate the proper equals method for all subclasses, but that may cause problems if we then extends the FLOJ class by a Java Class (which will be possible since we generate Java code).
But maybe it should be so, that a class is never equals to its subclasses? Returning equals between two instances that have the same data but different behavior (one instance has an overloaded method) is probably an error in the OO world...

Anyway, creating your own language is fun, I'm learning a lot and maybe I'll even end up with something usable!! I'll keep you posted on my progress.

Generator (with yield) in Java

2007-10-11T10:13:00.003+02:00

For some reason I was thinking about generators and the yield function (as found in Python). All this functional talk about infinite lists and such I suppose... So I had a go at implementing one in Java, which tied in nicely with my current reflexions about a java-based functional language and the need to provide concurrency out of the box. And another occasion to dabble with wait and notify in Java! After I've written it I found a similar looking implementation (but without the standard Iterator/Iterable interface support) and something on Google code that use bytecode intrumentation (why oh why). But still I like mine, that you can find here (and the unit test is here).
Basically the work is done in another thread and we call wait() when we do a yield. We conform to the Iterator and the Iterable interfaces, and we provide a stop method to actually stop the thread (say for infinite generators). Otherwise if the generator has not finished the thread goes on waiting forever. I've thought about trying to stop the thread when the generator is not used, but I don't know if it even possible with things like SoftReferences (The thread having a pointer to its runnable...).

My own Java-based functional language

2007-10-09T17:21:00.000+02:00

I've been toying for a while with the idea of creating my own Java based functional language, if only to be able to see first hand the challenges in creating a functional language. The language I envision has the following characteristics:
- syntax close to Java: contrary to JHaskell or CAL, I want something that looks like Java, not something totally different
- Java type system: use the normal Java OO type system: classes with at most one superclass, interfaces. Maybe I'll run into issues but for the moment...
- pure functional and monadic: what I love in Haskell is the clear delimitation between what's purely functional and what's subject to side effects (IO, etc...). I'd like to have that in my language. I know about Scala, but Scala lets you mix functional and imperative code, I want something pure (and I don't like to add new very un-Java like keywords like def)!
- Java integration: there is no point of being implemented in Java if we can't reuse the wealth of Java code and libraries out there. Probably I'll need a way to distinguish somehow functional Java code (like the String methods) and not functional code, and wrap access to non functional code in a monad.

Now that's an ambitious program!! I've started some parsing and conversion to Java source files (hey, if Java can do some of the type checking for me...), but it's going to be a long road...
Of course, if such a language already exists, I'd be delighted to hear about it!

Parser Combinator in Java

2007-09-19T13:05:00.000+02:00

Pffuii, I'm quite busy these days, hence the posting frequency has dropped. I'm in the process of playing with a functional Java based language (more on that in later posts, hopefully) and I decided to have a go at a little parser combinator in Java.
The end result works (I can parse things like Java class declaration) and looks a lot like parsec, but there is a lot of Java cruft that I'd like to remove, things that are hidden with the wonderful magic (I don't understand it well enough to call it otherwise) of Haskell monads.
For example:

try {
Character c=new Try(new Letter()).parse(input);
...
} catch (JCParseException jpe){
...

This code tries to match a letter in the input at its current position (the input variable) and does not consume any input if it fails. I don't like having to explicitly tell the combinator to parse the input, and I don't like using exceptions to indicate a parsing failure. Now, anybody has any great ideas?

The beauty of Haskell Combinators

2007-08-10T14:18:00.000+02:00

Pffiuu, it's been a while since I've posted here, but don't think I've been idle. In addition to my day job, I'm still hacking away at Haskell. And still loving it!! I just wanted today to express my pleasure (yes, pleasure) at working with some cool APIs and libraries that are out there.
I'm doing a bit of research into language parsing and I'm using Parsec for parsing and the HughesPJ Pretty Printing library (included with GHC) to dump the parsed AST and check that I can output what I parse and reparse it into a equivalent structure. I have to say, working with combinators (monadic or otherwise) is actually fun and easy. I've developed code parser before, for example using some Java parser generators, but coding your parser directly in your favorite functional language is really sweet. And of course, I like the way that similar concepts are expressed the same in the two libraries, even down to the function names: parens() in Parsec will parse something surrounded but parentheses and parens() in HughesPJ will write something in between parenthesis.
I had also a exhilarating ah ah moment when I discovered in the Parsec doc makeTokenParser that suddenly made it so easy to accommodate with spaces and pesky littls things of this caliber.
Thanks to all contributors to these libraries for making my functional programming evenings fun!

A very dumb neural network in Haskell

2007-06-01T13:12:00.000+02:00

Continuing my implementations of Michael Negnevitsky book on Artifical Intelligence in Haskell, I have tried to code a three layer neural network to implement exclusive or (Intelligence indeed...).

A neuron is made of its input weight and its threshold, and its output is calculated thus (apologies to all of you who gave me some tips about parameter order and newtype declaration, I haven't changed the code yet to follow your good advice!):


type Neuron= ([Value],Value)
type Value = Float

neuronOutput :: [Value] -> Neuron -> Value
neuronOutput inputs (weights,threshold) =
    let
        tot=(foldl (+) 0 (zipWith (*) inputs weights)) - threshold
    in
        1 / (1 + ((2.7182) ** (-tot)))

A three layer network whose input layer does nothing but fire its input to the midle layer can be modelled by to lists of neurons, and I use a TestCase type to keep track of inputs and expected outputs

       
type Network = ([Neuron],[Neuron])
type TestCase = ([Value],[Value])

Running the whole network with some input values is easy:


exec ::  Network -> [Value] -> ([Value],[Value])
exec (hidden,output) inputs =
    let
        hiddenOutputs = map (neuronOutput inputs) hidden
        outputOutputs = map (neuronOutput hiddenOutputs) output
    in
        (hiddenOutputs,outputOutputs)

The meat is all in one function, that calculates the output of the network for a given test and adapts the weights accordingly, first for the output layer then for the hidden layer:


defaultLearningRate::Value
defaultLearningRate=0.1

step ::  (Network,Value) -> TestCase -> (Network,Value)
step ((hidden,output),err) (inputs,expected)=
    let
        (hiddenOutputs,outputOutputs)=exec (hidden,output) inputs
        errorOutputs= zipWith (-) expected outputOutputs
        gradientOutputs=zipWith (\o d->o*(1-o)*d) outputOutputs errorOutputs
        newOutputs=zipWith (\(ws,t) g -> ( zipWith (\h w->w+ (defaultLearningRate*h*g)) hiddenOutputs ws ,t+(defaultLearningRate*g*(-1)))) output gradientOutputs
        weightsByHidden= transpose (map fst output)
        gradientHidden =zipWith (\o ws-> o * (1-o) * (foldl (+) 0 (zipWith (*) ws gradientOutputs))) hiddenOutputs weightsByHidden
        newHidden=zipWith (\(ws,t) g -> ( zipWith (\h w->w+ (defaultLearningRate*h*g)) inputs ws ,t+(defaultLearningRate*g*(-1)))) hidden gradientHidden
        newErr=err + (foldl (+) 0 (map (^2) errorOutputs))
    in ((newHidden,newOutputs),newErr)

The result is a new adapted network and the sum of squared errors.

An epoch is a set of tests:


epoch :: Network -> [TestCase] -> (Network,Value)
epoch network= foldl step (network,0)

We can then repeat epoch till the sum of squared errors for an epoch is less than the threshold for convergence:


errorConvergence::Value
errorConvergence=0.001

run ::     Network -> [TestCase] -> Int -> (Network,Int,Value)
run network allInputs epochNb=
    let (newNetwork,delta) = epoch network allInputs
    in (?) (delta <= errorConvergence || epochNb>225)
            (newNetwork, epochNb,delta)
            (run newNetwork allInputs (epochNb+1))
   
(?) :: Bool -> a -> a -> a
(?) True a _  = a
(?) False _ a = a

(Is there a standard function for (?)? I got tired of if then else...)

Then I test my code with the same start network as in the book:


test = do
    let n = ([([0.5,0.4],0.8),([0.9,1.0],-0.1)],[([-1.2,1.1],0.3)])
    let (n',e,err) = run n [([1,1],[0]),([0,1],[1]),([1,0],[1]),([0,0],[0])] 1
    print (n',e,err)
    return ()

And I get a mixed feeling about the result: the network does converge, and then it does work, but after something like 54000 epochs, while in the book it supposedly take 224! The book gives the result for the first test and I'm getting exactly the same thing, but it looks like my network learns a lot slower than it should. I would suppose I have a subtle error somewhere, but for the life of me I can't find it... If it jumps at anybody, please let me know. My hope now is that when refactoring the code and following the books advice on improving the performance I will find the bug!!

I have tried with random networks:


network :: Int -> Int -> Int-> IO Network
network inputNb hiddenNb outputNb= do
    a<-randomNeurons inputNb hiddenNb
    b<-randomNeurons hiddenNb outputNb
    return (a,b)
   
randomNeurons:: Int -> Int -> IO [Neuron]
randomNeurons nbInput nbNeurons= replicateM nbNeurons (randomNeuron nbInput)
   
randomNeuron:: Int -> IO Neuron
randomNeuron nb= do
    w<-replicateM nb (randomWeight nb)
    t<-randomWeight nb
    return (w,t)
   
randomWeight:: Int -> IO Value
randomWeight nbInput= do
    let interval= randomR ((-2.4)/(fromIntegral nbInput),2.4/(fromIntegral nbInput))
    getStdRandom interval

But I don't get better results...

A perceptron in Haskell

2007-05-25T14:12:00.000+02:00

After reading Michael Negnevitsky book on Artifical Intelligence, I started playing with some of the algorithms he gives in Haskell. So my first endeavour is writing a simple perceptron (a one neuron neural network) to perform AND or OR logical operations. Next I'll start a full multi layer network to do much more complex operations like exclusive OR (-:.
Once again I was struck with the simplicity of the Haskell code necessary to implement such functionality (and I'm no Haskell expert yet). You think about how you going to implement it, the type checker tells you when you're doing something stupid, and by the time everything is written and compiles it kinda works.
The core method is calculating the output of the neuron, given the inputs, the weight of each input and the threshold value for that neuron:


neuronOutput :: [Float] -> [Float] -> Float -> Float
neuronOutput inputs weights threshold =
    let
        tot=(foldl (+) 0 (zipWith (*) inputs weights)) - threshold
    in
        if tot >= 0
            then 1
            else 0

Using zipWith we multiply each input with its weigth, calculate the sum with fold and remove the threshold. A negative result yields 0, else it yields 1.

Adjusting the weights to adapt to the error of a test case is no more difficult:


defaultLearningRate::Float
defaultLearningRate=0.1

adjustWeights :: [Float] -> [Float] -> Float -> Float -> [Float]
adjustWeights inputs origWeights expected actual =
    let
        e=expected - actual
        delta (i,w) = w + (defaultLearningRate * i * e)
    in
        map delta (zip inputs origWeights)

We modify the weight by applying to each weight a delta that is dependant on the input weight, the error ratio and the learning rat

A single step is made of calculating the output of the neuron and adapting the weights, given the expected value:


defaultThreshold::Float
defaultThreshold=0.2

step :: [Float] -> [Float] -> Float -> [Float]
step inputs weights expected=
    let
        o = neuronOutput inputs weights defaultThreshold
    in
        adjustWeights inputs weights expected o

Then an epoch applies the step function to each different input sets. The hardest part is calculating the average of all deltas to find if the weights have changed at all during the evaluation of all the test cases


epoch :: [([Float],Float)] -> [Float] -> ([Float],Float)
epoch allInputs weights=
    let
        f w (inputs,expected) = step inputs w expected
        newWeights=foldl f weights allInputs
        delta= (foldl (+) 0 (map abs (zipWith (-) newWeights weights))) / (fromIntegral $ length weights)
    in (newWeights,delta)

A training run is only launching the epoch function repeatedly till the delta is zero:


run ::     [([Float],Float)] -> [Float] -> Int -> ([Float],Int)
run allInputs weights epochNb=
    let (newWeights,delta) = epoch allInputs weights
    in if delta == 0
        then
            (newWeights, epochNb)
        else
            run allInputs newWeights (epochNb+1)

We generate the initial weights in the IO monad, passing the number of weights needed:


initialWeights :: Int -> IO [Float]
initialWeights nb = do
    let interval= randomR (-0.5,0.5)
    (replicateM nb (getStdRandom interval))

Putting it all together:


test :: IO()
test = do
    w<-initialWeights 2
    let (ws,i)=run [([0,0],0),([0,1],0),([1,0],0),([1,1],1)] w 1
    print (ws,i)
    return ()

test find out the proper weights to implement AND and print them out, with the number of epochs needed to reach the result. You can then call neuronOutput with the resulting weights and your input: you have a AND logical gate!

As usual, any pointer to how to improve the code is welcome!

Javascript and Hibernate, hand in hand

2007-05-16T13:38:00.000+02:00

Some time ago I had written a little prototype that was quite interesting, but I never pursued it. Maybe I should try to get some people interested and develop it further...
The idea was to use Rhino to have a Javascript-based framework on the server, thus using the same language on the server and on the client, like ASP did in the good old days. I was also thinking of having the source code for custom applications stored in the database, instead of having it in source files, but that's another story. That explains why the examples below are about "applications" and "components"
The first component is Hibernate connectivity. My framework lets you:
1. Dynamically configure Hibernate mapping for Javascript defined classes. For example, if you wanted to define a Web Application with several attributes and a reference to the components that it defines:


function Application(){
 this.displayName='';
 this.urlName='';
 this.defaultUrl='';
 this.id=-1;
 this.components=[];
}


Application.storage(
 {
 id:
  {name:'id',column:'ID',type:'long'},
 properties: 
  [
   {name:'displayName','not-null':true},
   {name:'urlName','not-null':true,unique:true,'unique-key':'APPLICATION_URL'},
   {name:'defaultUrl'}
  ], 
 references:
  [
   {'list':
    {name:'components',table:'APPLICATION_COMPONENT',
     details:[
      {'key':{column:'APPLICATIONID'}},
      {'list-index':{column:'idx',base:'0'}},
      {'many-to-many':{'column':'COMPONENTID','class':'Component'}}
     ]
    }
   }
  ]
 }
 );

Note that storage() is a function of the Function object which takes a Javascript structure that mimicks the Hibernate XML.

2. Access Hibernate functionality through a "hibernate" host object. For example, a JSUnit method testing the addition of a new "Application" object:


var appId;
var appUrlName;

function testAdd(){
 assertNotUndefined(hibernate);
 var app=new Application();
 assertNotUndefined(app);
 app.displayName="TestAppJS"+new Date();
 app.urlName=app.displayName;
 appUrlName=app.urlName;
 hibernate.save(app);
 hibernate.commit();
 assertNotNaN(app.id);
 assertTrue(app.id>0);
 appId=app.id;
}

The hibernate object lets you list objects, get a particular object, modify or create instances, etc. I have implemented a Hibernate EntityTuplizer and an subclass of Rhino's NativeArray that encapsulate a PersistentList, so that multiples references can be seen as Javascript arrays.

3. Query the database either through the HQL language:


var apps=hibernate.query("from Application where urlName=?",["TestAppJS"]);

or through a pure Javascript function based mechanism:


var apps=hibernate.query("Application",function(app){
 return app.urlName=="TestAppJS";}
);

That I think is very cool. In Rhino I can access the source code of the function that is passed as the second argument to query, and transform it into HQL. I haven't implemented as much as I could, of course, but it works with simple queries. At least there is no other syntax to learn to do a query, it's basically the equivalent of a filter function in functional language.

Then I had started to develop a web framework (probably the millioneth Java web framework by now) with Javascript objects to generate HTML and a mapping between urls and javascript functions, and where one URL could chain several functions, but I haven't gone very far.

Now if somebody feels there's some value in any of that I can sure post the code and we can move it forward. Anything that can show that Javascript is a cool language to run in a JVM in a worthwhile enterprise (-:

Poor man's JavaFX Script is, er... Javascript

2007-05-11T13:32:00.000+02:00

Sun has release its own RIA framework called Java FX. It provide JavaFX Script, which is a language based on Java whose goal is to develop quickly graphical applications with the power of Java. Is it touted as competition to Flex that I have mentioned before. It is safe to say that it hasn't been received with a great deal of enthusiasm. The fact is, I don't want to learn another language again to do my Web 2.0 application. And I do believe that Fielding has a point when he reflects that applets break visibility and raise the barrier of entry for web developers. Also, I'm usually all in favor of static typing and compilation, but for GUIs I'm not too sure. When you want to tweak your borders or your layouts you probably happy enough with a quick cycle of change code/refresh page. So why don't we use Javascript and Java Applets together instead of inventing yet another technology?
Ok, it's time you get over your irrational distrust of Javascript. A multi paradigm (object, functional) language available for free in every browser!! And no, I don't suggest we go on using the HTML DOM or things like that. That's why we need applets.
Applets? I must be mad! Applets have not worked on the web. But what I suggest is not to develop one applet for each RIA application, but only develop one Applet that everybody could use, much like a plugin, that would bridge the gap between the JavaScript world and the Java/Swing world. If it has been done, please let me know. As for me, I've put together a prototype in an hour that works fine:

The JavaScript code constructs an object tree that represents the GUI.
The object tree is serialized to JSON and passed on to the applet
The applet deserializes the JSON and constructs a Swing GUI by translating the object tree
The object tree can contain handlers that are the names of Javascript functions
The applet calls the javascript functions on Swing events
The JavaScript event handlers update the GUI tree and notify the applet with the changes.

A very simple example:


<html>
<head><title>JSApplet</title>

<script language="Javascript">

var helloButton={type:"button","id":"helloButton","text":"hello","action":onClickHello}
var frame={"contents":
 [
  helloButton
 ]
 };


function onClickHello(){
 helloButton.text=helloButton.text.toUpperCase();
 updateJS([helloButton])
}

function startJS(){
 var msg=document.getElementById("JSApplet").setContent(toJSON(frame));
 if (msg){
  alert(msg);
 }
}

function updateJS(objs){
 var msg=document.getElementById("JSApplet").updateContent(toJSON(objs));
 if (msg){
  alert(msg);
 }
}


toJSON=function(arg) {
    
    switch (typeof arg) {
    case 'object':
        if (arg) {
            if (arg.constructor == Array) {
                var o = '';
                for (var i = 0; i < arg.length; ++i) {
                    var v = toJSON(arg[i]);
                    if (o) {
                        o += ',';
                    }
                    if (v != null) {
                        o += v;
                    } else {
                        o += 'null,';
                    }
                }
                return '[' + o + ']';
            } else if (typeof arg.toString != 'undefined') {
                var o = '';
                for (var i in arg) {
                    var v = toJSON(arg[i]);
                    if (v != null) {
                        if (o) {
                            o += ',';
                        }
                        o += toJSON(i) + ':' + v;
                    }
                }
                return '{' + o + '}';
            } else {
                return;
            }
        }
        return 'null';
    case 'unknown':
    case 'undefined':
    case 'function':
     return arg.name;
    case 'string':
        return '"' + arg.replace(/(["\\])/g, '\\$1') + '"';
    default:
        return String(arg);
    }
}

</script>
</head>
<body onload="startJS();">
<applet id="JSApplet" alt="JSApplet" code="fr/moresmau/jp/jsapplet/JSApplet.class" codebase="bin" mayscript="mayscript" height="100" width="100"></applet>



</body>
</html>

This web page creates a simple "Hello" button that when clicked changed its text to upperCase(). It demonstrate the two ways communication between the Javascript code and the applet (note how the function is serialized in JSON just with its name, that's the only thing we need). So the Javascript can deal with a model that is a slightly abstracted Swing model (no more hair pulling between different browsers). I stress again, the applet is a generic applet that deals with the link between the JSON objects and the Swing layer. Of course the applet could offer other services: simple API to call back the server (AJAX from the applet), etc...
Then the applet could be packaged with a cut down version of the JRE (only the classes really needed) as a plugin to do rich Swing application from JavaScript...
And voilà! A scriptable RIA platform!

Haskell code, Java UI

2007-04-20T13:28:00.000+02:00

It's been while since I've posted anything, mainly because work crises and holidays have kept me busy or away from my keyboard. I'm still playing with Haskell and user interfaces, I've now developed the same Maze game in HGL, Flex with a Haskell web back end, and WXHaskell. I've looked at trying to implement some declarative syntax that would simplify WXHaskell code but haven't gone very far, I think I'm trying to run in Haskell before I can walk straight!
Anyway, I was looking at Business Objects' Quark framework based on the CAL language. You can find more details here, but basically CAL is a Haskell-inspired impure functional language implemented in Java, that can use Java classes and methods. The impurity comes from not using monads to access Java, if I understand well, which means that you can declare pure functional methods that in fact access Java objects, that can do all kind of stateful voodoo under the covers. While I'm not sure this is really A Good Thing, having Haskell and Java intertwined brings some interesting ideas in the realm of Haskell UIs. Swing and Java are now mature, reasonably fast and cross platform. If I can get my pure Haskell code to talk to a Java GUI, I could get quite happy indeed...

A Haskell GUI in Flex

2007-03-30T15:18:00.001+02:00

In an earlier post I mentioned a little Haskell game I developed where you had to navigate in a maze. I used HGL as the GUI framework, and noted it wasn't very clear what was the functional GUI framework of choice for Haskell. So I decided to give Flex a go and to hook it up with my Haskell back-end. I'm quite please with the result, I have to say.
The setup is as follows:
1) I've implemented a very simple single-threaded HTTP server in Haskell, using as an inspiration code that I found here and there (using Parsec for parsing HTTP messages, etc.). The server can server static HTML and SWF files as well as having a dynamic component: you can register Haskell methods that take a JSON object as a parameter, some internal state and return a new JSON object and the new state (yes I know, there is a monad for that...). I used the code here for JSON in Haskell. I thus expose a method that generates a Maze and a method to move to a cell, validating that there is no wall in between the current cell and the target cell and checking to see if we reached the exit.
2) I use Flex and ActionScript to create the GUI. The compiled SWF file will be served by the Haskell server and will call it back through asynchronous calls. I was a bit taken aback when I realized that Flex doesn't come with JSON support out of the box, but luckily there's a library for it. So you connect via your browser to a page on the server that embeds the SWF, Flex ask for the maze data to the server and displays it, and let you navigate using the keyboards or the mouse.
And that's it! So I have a nice, platform independent GUI and functional business code. I was thinking, who needs a special GUI framework for the desktop when an application could be made easily with a very lightweight local HTTP server and a web based interface? I even thought that Flex was a bit too much: why do I need a GUI technology that requires compilation when I can do nice stuff with pure HTML/Javascript frameworks like Dojo? So I can use a UI technology that is purely for UIs, and a functional language with the goodies Haskell provide for all the back-end, computation-heavy work.
The code for Haskell is here (start from Framework.hs), the Flex code is here.

Dynamic class behaviour in Java

2007-03-23T13:39:00.000+01:00

In a previous post, I was using annotations to generate method pointers in Java. The functions objects were generated in a separate class file. However, if you suppose that a modern IDE would be able to generate all that code for you when you create a method, you could have the function object in the same class file as the method code itself. Since Java allows you to have a field and a method with the same name, everything works fine.
In that previous post the function object referenced the actual method in the original class file. What if we did the opposite, that is to say, the actual code is in the function object, and the method merely delegates? Here is a trivial example:


public class StringHelperDynamic {

    public String toUpper(String a){
        return toUpper.run(a);
    }
   
    public Function1 toUpper = new         Function1 (this){
            public java.lang.String run(java.lang.String a) {
                return a.toUpperCase();
            }
        };
   
}

Here, we have a toUpper method that takes a String and return a String, and a toUpper function object that we can pass around, perform currying on, etc... the interesting twist of this architecture is that the toUpper field can be final or not. If not, we can then change it at runtime, and hence change the behaviour of the class at runtime, while still calling the same methods. Who needs a dynamic scripting language (-: ? To the silly example:


StringHelperDynamic me=new StringHelperDynamic();
me.toUpper=new Function1(me){
    public java.lang.String run(java.lang.String a) {
        return a.toLowerCase();
    }
};

assertEquals("hello",me.toUpper("HELLO"));

OK, breaking the contract of the method is NOT what this is about, but you get the gist of it. The code for the Function1 class can be found here.

VList in Java

2007-03-16T14:10:00.001+01:00

Two days ago I was reading this article about LISP like lists implemented in Java,
which reminded me I had written an implementation of a VList in Java. It can be found
here, along with a unit test.I make no garantee this is the best or fastest implementation possible!

It implements: get(n), first (head in Haskell), rest() (tail in Haskell), size(), iterator() and map(Mapper), where Mapper is the mapping interface I mentioned earlier.

It's using Java Generics of course. Am I right in thinking there is no way with Java generics to get rid of the compilation warnings and of the necessity of passing the Class object in the code that creates a new array of the generic type?

Java properties with no language changes (oh no!)

2007-03-09T15:55:00.000+01:00

There's been a lot of discussions on the web about Java and the support for bean properties. People would like to have a mechanism that relies less on java bean conventions (reflection on method names, find getters and setters) and more on explicit language constructs. The goal is to simplify bean and property handling, have stronger type checks, be able to reference the property and not its values, etc... Most proposals involve additions to the Java syntax. I don't want to add to the confusion, but I just played around with what we have now in Java 5.
What we could have is instead of defining classes directly with fields in them, we could have them reference Property objects instead. Property objects carry the type information and the value itself. An inheritance hierarchy can provide for read-only, write-only and read-write hierarchy. Using public final fields we can have an access syntax that is not the same as the current getter and setter methods, but pretty close.
Example:


public class PersonBean {
    public final IFullProperty name=new NonNullProperty();
   
    public final IReadProperty id;
   
    public final IFullProperty email=new FullProperty(){
        @Override
        public void set(String value) {
            if (value.indexOf('@')==-1){
                throw new IllegalArgumentException("email does not contain @");
            }
            super.set(value);
        }
    };
   
    public PersonBean(int id,String name){
        this.name.set(name);
        this.id=new ReadOnlyProperty(id);
    }
}

Of course, the generic madness means the type information has to be typed twice, but when generics syntax is cleaned up it'd be a lot more readable. This bean says that it has a changeable name property, a String that cannot be null. It has an id integer property that cannot be changed. Finally it has an email property that has ad hoc constraint checking. The Property interfaces and its sub interfaces define a get() and a set() method so accessing the fields becomes:


bean.getName() -> bean.name.get()
bean.setName(name) -> bean.name.set(name)

The property change listener can be attached to the property object so that the bean doesn't have to have any line of code related to listeners. Everything can be done via static methods:


static void addPropertyChangeListener(Object bean,PropertyChangeListener pcl);

to add a listener on all properties


static void addPropertyChangeListener(Object bean,PropertyChangeListener pcl,IProperty p);

to add a listener on a specific property. Note that since we're taking a property object, and not a property name for example we cannot register a listener on a non existing property. The property does not have a back link to its bean (to avoid disgracious constructors with this), so we need to have both the bean and the propert.
Example:
addPropertyChangeListener(bean,listener,bean.email)

If anybody's interested, all the code with unit tests is available here.

Haskell User Interface???

2007-02-02T12:23:00.000+01:00

It's been a long time since I've last posted, the fault to Christmas and holidays. In the meantime though I have done some more Haskell. I have used Prim's algorithms described here to generate a random maze. That forced me to use Random to generate the maze and HGL to display it, and allow the user to navigate from the entrance to the exit with direction keys. Works great, but I was looking at Haskell UIs to see what I could use, and there doesn't seem to be a consensus on a functional GUI library in Haskell. HGL is fairly low level and limited, wxwidgets ports have an imperative air about them and some of the ground breaking frameworks still seem to be experimental (and to be honest, I don't think I'm ready to deal with arrow-based frameworks yet, I'm still struggling with the monads...). So maybe the solution is elsewhere? I've seen in the Java world people talk about Flex, has anybody done anything integrating a Flex front end with a Haskell back-end?

Adventures in Haskell: the Parsec magic weapon

2006-12-15T13:53:00.000+01:00

Following suggestions, I have rewritten the parsing code using Parsec. I have to say, it was a good experience. Not everything worked first time, but I feel the code is a lot clearer that doing everything myself line by line. I've compared the two source files: DataAdvParser.hs and the new DataAdvParsec.hs. There's more lines in the Parsec based files because I've had problems with the normal Haskell layout, but the file is overall smaller. Moreover, I have discovered ambiguities and bugs in the initial version when I had to get Parsec to work properly. And of course looking at the Haskell code you actually understand the grammar of the input file...
I don't feel at the moment I'm getting better in Haskell: using Parsec just meant using a library API but didn't mean much in terms of learning the language or the underlying technologies (monads, etc...). I'm thinking maybe writing a little DSL for my adventure game data would help, but I haven't yet found a good tutorial. Ideally I'd like to get to something that's similar to what's been done for HTML or SQL, but for a very simple language...
Anyway, I'll see, next steps would be to write maybe an automated game world generator so that I could play a new game every time...

An infinite list in Java

2006-12-08T15:10:00.001+01:00

In Haskell you can define infinite lists by providing, for example, the arithmetical properties of a suite. Lazy evaluation means only the elements needed by the program are calculated. Now, I was looking at doing the same thing in Java, and there's not too much to it.
An infinite list is an Iterable over a generic type (called E in my code). The base class is abstract, and you need to provide the implementation of a getNext() method, that provides the next element in the list. The list is backed by an ArrayList, so that values are only calculated once. There is of course no size() method, but the size of the underlying array list tells you how many elements have been calculated. Note that each calculated value being stored in memory, you cannot go too far in a list with this system. But hey, we're playing here.

For example this is how you can implement the Fibonacci suite, using the code for InfiniteList here:

InfiniteList il=new InfiniteList(){
@Override
protected Integer getNext() {
switch (this.getData().size()){
case 0:return 1;
case 1:return 1;
default: return this.getData().get(this.getData().size()-1)+this.getData().get(this.getData().size()-2);
}
}
};

You then use the iterator() method to get all the results you need. Don't forget to stop at some stages. The getNext() method is called by the calculateNext() method on the base class:

private void calculateNext(){
E next=getNext();
data.add(next);
}

which in turn is called by the ensureCalculated method, that checks that we have the value at a particular index:

private void ensureCalculated(int index){
while (index>=data.size()){
calculateNext();
}
}

So the standard list get method becomes:

public E get(int index) {
if (index<0){
throw new IllegalArgumentException("index<0");
}
ensureCalculated(index);
return data.get(index);
}

You can then have the methods Haskell provide you, for example, take and map. Take returns a sublist made of the nth first elements of the list and map gives back another infinite list calculated from the first by applying a function to each element, as required. We can use the following interface:

public interface Mapper {
E2 map(E1 initial);
}

And the map method is:

public InfiniteList map(final Mapper mapper){
return new InfiniteList(){

@Override
protected E2 getNext() {
return mapper.map(InfiniteList.this.get(getData().size()));
}
};
}

We can also have another infinite list that is a filtered list from the original, using the same interface as above with the E2 generic type a boolean. Watch out, though: if the filter returns false all the time, a call to next() might never return...

The full code can be found here for the InfiniteList class, here for the Mapper interface. Sample code in the form of unit tests are here.

Adventures in Haskell: parsing the game world

2006-12-05T09:28:00.000+01:00

It's been a while since my last post in Haskell, as it's hard to find the time when I spend 40 hours a week doing Java, and there's little people to take care of in the house... I have been working on my adventure game, still, mainly to separate the data from the actual code. After a fews days of tinkering, I have it working, yoohoo!
Of course, having to write a simple parser was a bit challenging given my limited knowledge of the language, and I wanted to stay with simple constructs. Hence,
I don't use any monadic stuff, I just read the data file and deal with a list of Strings, one String per line, having each method take the list and send back the remainder of list. Promise, I will look into a state monad! And I know there some advanced parsing techniques in Haskell, I mean, more advanced than reading lines and calling words on it...
I had fun with lazy IO, as any newbie Haskell programmer: using getContents seemed sometimes not to give me any lines back, until I understood than evaluating the result after closing the file handle was not the smart way to do things...
So much for the "once it compiles, it works" mantra that I was chanting last time: when parsing a file, the content matters, too, not only the code, so I had trouble solving all the unmatched pattern errors I got. I found GHC more helpful than Hugs for error messages: an error that got me scratching my head for a while in Hugs got solved as soon as I ran the same code in GHC.
I tried as well Hugs.Observe to debug my program, but I fear my grasp of lazy evaluation is too limited yet: sometimes I was sure I was getting into a method but observe was not reporting anything...
Anyway, if anybody is interested in the code, the parser code is here, and an example of a data file is here. The structure is simple enough, blank lines are used a separator and on most lines the first couple of words are keywords (item codes, location codes, etc...). The adapted game code is here (note that the path to the data file is hard-coded here)
As usual, any comments and suggestions welcome!

Higher order functions in Java with an annotation processor factory

2006-11-24T15:37:00.000+01:00

I’m continuing my explorations of both Java 5 features like generics and annotations and of functional programming constructs. A lot of Java discussions on the web these days are about closures: writing quick anonymous methods without having to wrap them in a abstract class (think Runnable, Comparator, …). Now, closures can also be related to higher order functions, that is to say functions considered as objects in their own right. If you can reference a precise method inside an instance instead of just having to deal with interface, you can have a type of closures. For example, instead of passing a Comparator object you could pass a Function object, meaning a function that takes two T arguments and returns an integer.
To illustrate this point, let’s take an example.
You have a simple Sale class:

public class Sale {
private String customer;
private Date dateOfSale;
private float amount;

public Sale(String customer, Date dateOfSale, float amount) {
…constructor, getters and setters…

And you have a SaleManager class that encapsulates a list of sales (think retrieved from a database, etc…) and gives you a method to retrieve all sales sorted by what you want:

public class SalesManagerClassic {
private List sales;

public SalesManagerClassic(List sales){
this.sales=sales;
}

public Sale[] listSales(Comparator comp){
Sale[] ret=new Sale[sales.size()];
sales.toArray(ret);
Arrays.sort(ret,comp);
return ret;
}

}

So far so good. Now you define a SaleComparator that will provide you all kinds of comparators:
public class SaleComparatorClassic {

public Comparator compareByCustomer(){
return new Comparator(){
public int compare(Sale o1, Sale o2) {
return o1.getCustomer().compareToIgnoreCase(o2.getCustomer());
}
};
}

public Comparator compareByAmount(){
return new Comparator(){
public int compare(Sale o1, Sale o2) {
return (int) (o2.getAmount()-o1.getAmount());
}
};
}

}

As you can see, creating a comparator each time is tedious. Closure and higher order functions aim at simplifying the important code bits so that the actual strategy (how do we sort the sales) is not buried in all the boilerplate code. This is how we could use that class:

List l=new ArrayList();
…
SalesManagerClassic smc=new SalesManagerClassic(l);
SaleComparatorClassic scc=new SaleComparatorClassic();
Sale[] s=smc.listSales(scc.compareByCustomer());

A Function object in Java is something like that, roughly:

public abstract class Function {
private Object instance;

public Function(Object instance){
this.instance=instance;
}

public Object getInstance() {
return instance;
}
}

So it encapsulates the instance on which the function will be run. We’re not going out of the OO world right now.
Then generics in Java do not (if they do, drop me a line) allow you to specify that a class can take any number of generic type in its definition, so we’ll have to define a concrete class for all possible numbers of parameters:

public abstract class Function0 extends Function {
public Function0(Object instance){
super(instance);
}

public abstract R run();

}

The code above represents an object method that takes no argument and its return type is the generic type R (which could be Void if the method return nothing). Subclasses need to provide the actual implementation of the run method. And a function taking one parameter will be:

public abstract class Function1 extends Function {

public Function1(Object instance){
super(instance);
}

public abstract R run(P param);

public Function0 curry(final P param){
Function0 f=new Function0(getInstance()){
@Override
public R run() {
return Function1.this.run(param);
}
};
return f;
}

}

Here P is the type of the first parameter. For fun I added the curry function: if you know the first parameter but you don’t want to run the function right now, you can create a Function object with no argument that can be run later.

So what would that mean for our sales manager? This is what the code can look like:

public Sale[] listSales(final Function2 func){
Sale[] ret=new Sale[sales.size()];
sales.toArray(ret);
Arrays.sort(ret,new Comparator(){
public int compare(Sale o1, Sale o2) {
return func.run(o1, o2);
}
});
return ret;
}

The boiler plate for the comparator has been moved in one place (of course we still need it, we’re not writing our own version of Java where standard API calls takes our Function objects right now…). The Sale comparator is where the gain is made:

@FunctionAnnotation
public int compareByCustomer(Sale o1,Sale o2){
return o1.getCustomer().compareToIgnoreCase(o2.getCustomer());
}

@FunctionAnnotation
public int compareByAmount(Sale o1,Sale o2){
return (int) (o2.getAmount()-o1.getAmount());
}

The FunctionAnnotation is a simple method level annotation. This tells us: I want to be able to refer to these methods as Function objects. Note that my code for the moment does NOT deal with methods with the same name and different argument list. We’ll worry about that later… Notice for now how the methods are much simpler and how what they do is clear.

Now, what we need to do, is to do automatically the translation between simple methods with the FunctionAnnotation and Function<…> objects that we can pass to our SalesManager. For this we’ll use an annotation process factory. This factory will run on the source code for SaleComparator, and will generate a SaleComparatorFunctions class that will have methods giving you the Function<…> objects:

public static Function2 compareByAmount(final fr.moresmau.jp.func.samples.SaleComparatorFunctional instance){
return new Function2 (instance){
public Integer run(fr.moresmau.jp.func.samples.Sale param0, fr.moresmau.jp.func.samples.Sale param1) {
return instance.compareByAmount( param0, param1);
}
};
}

public static Function2 compareByCustomer(final fr.moresmau.jp.func.samples.SaleComparatorFunctional instance){
return new Function2 (instance){
public Integer run(fr.moresmau.jp.func.samples.Sale param0, fr.moresmau.jp.func.samples.Sale param1) {
return instance.compareByCustomer( param0, param1);
}
};
}

Aren’t we happy this is generated for us ? Then we can just code our calls like that:
List l=new ArrayList();
…
SalesManagerFunctional smf=new SalesManagerFunctional(l);
SaleComparatorFunctional h=new SaleComparatorFunctional();
Sale[] s=smf.listSales(SaleComparatorFunctions.compareByCustomer(h));

That code is about the same complexity as before. Of course using another class for the function wrapping make the code not as clear as it should be. In a later post I’ll look at alternatives, so for the moment bear with me.

So we have obtained higher order functions from a standard Java class, and we can use these functions instead of comparator objects.

Now of course I haven’t talked about the annotation process factory yet. There’s nothing magic there. We need to implement an AnnotationProcessFactory and to say we want to do something when we encounter our annotation:

public Collection supportedAnnotationTypes() {
return Arrays.asList("fr.moresmau.jp.func.FunctionAnnotation");
}

And we need to implement a AnnotationProcessor that will generate a Functions class for every class that as at least one method with the FunctionAnnotation. For that we need to use the createSourceFile method of the Filer object provided by the environment to create the function file.
Running the factory is done through the JDK apt tool:

apt -s src -factory fr.moresmau.jp.func.FunctionProcessorFactory -d bin -cp bin src/fr/moresmau/jp/func/samples/SaleComparator.java

The code for the Function objects and the AnnotationProcessFactory can be found here, the Sale example here, with unit tests.

Design By Contract in Java using Annotations, Scripting and Aspects

2006-11-17T10:03:00.002+01:00

When you write a Java interface, you define a contract. This contract is made of methods and their signatures. Then, in the implementation code, you can write statements to check that the input parameters are not null, that return values are within a correct range, etc. The problem there of course is that if the implementation doesn't do these checks, or not properly, bugs
may creep in and they may be hard to find. "Design by Contract" is a set of techniques that
let you specify conditions on method parameters, return values, etc, in such a way that they
can be associated with interfaces and that the execution of these statements is assured by the
runtime system, freeing the developer from the tedious tasks of throwing
IllegalArgumentExceptions or assertions.

To implement such techniques, you can use some off the shelf frameworks, like jContractor, but it's easy to roll out your own. You need only three components:

you need to be able to attach conditions to classes and methods, without cluttering the code. For this, Java annotations are perfect
since we're not in Java code proper, we need a way to run our conditions, so we need a language for them. Well, why have another language, just use a Java scripting languages like Beanshell.
finally, the Beanshell code needs to be evaluated at runtime before or after methods calls. This is what aspect are for, right? We'll use AspectJ for that.

So, let's see a concrete example. Suppose we implement a Stack (yeah, right, I don't have a lamer example).
We start with the push method:

public interface IStack {
void push(T o);
}

Ok, but we don't like to have null values in this stack. So we want to specify in the interface definition that the first parameter cannot be null:

@ContractMethodAnnotation (
pre = {"p0!=null"}
)
void push(T o);

We cannot access the names of the method parameters at runtime, so we use a simple convention: p0 is the first parameter, etc. You can also define post conditions that can access the return value using the name "return".

Notice that we specify conditions on the interface, but that we will want the checks to apply to all its implementations.

Conditions can access methods and variables of the class you define them on, but be careful, it's easy to create circular references where two method call each other:

boolean isEmpty();

@ContractMethodAnnotation (
post = {"result>=0"}
)
int size();

It would be tempting to define a condition that isEmpty is true if size()==0 and that size()==0 if isEmpty is true... What you can do is define a class invariant condition: a condition that is always true of an object (even if it can transiently be false, it is true before and after public method invocation).

@ContractClassAnnotation (
invariants={"(isEmpty() && size()==0) || (!isEmpty() && size()>0)"}
)
public interface IStack { ... }

The implementation of the aspect is not very complicated. We define pointcuts for public methods that will check class invariants and for annotated methods that will check pre and post conditions. We use the BeanShell Interpreter object to parse and run the condition code. Thanks to the importObject Beanshell command we can run the code as if all methods referenced by the annotation code referenced the current instance.

When we encounter a failure, we can either use an assertion or throw a RuntimeException.
This can be set through the ContractAspect.setUseAsserts method. For fun, I have defined a way to set it through a bean shell configuration file, that is loaded by another aspect, that executes when the ContractAspect is statically initialized.

The code can be found here: ContractAspect.aj is the aspect implementation, ContractClassAnnotation.java and ContractMethodAnnotation.java are the definition of the annotations. SettingsAspect.aj is the aspect that reads the configuration file and turn assertions on or off. An example of a configuration file is here. As you see, it's only one line of Java code.

The Stack example can be found here, with an implementation and a unit test.

My first Haskell adventure game!

2006-11-10T16:36:00.001+01:00

After years of coding mainly in Java, I have taken the plunge and looked
into a pure functional language. Haskell is quite popular these days, so I decided to read a book
on it and play around with it myself.
I got a bit frustrated at the maths or geometry examples, when I got to this great article Casting spels in LISP that uses the medium of a text based adventure game to teach the basic notions of LISP.
So I coded a very little (400 lines with data and documentation) text-based game in Haskell.
The code can be found here (You need a little extra module found here). In my enthusiasm I even added some Haddock documentation here.
The main features are:
- use of the IO monad to get user input and give back feedback
- use of the IO monad to save and load gaves
- use of the Data.Map structure
- use of the Show and Read classes to serialize and deserialize a Map structure
- pure functional handling of state: every change in the game state recreates an instance of the
GameState object. Yes, one day I'll look into the State monad to simplify these functions

My first impressions of Haskell:
- the syntax is quite hard to grasp, with loads of different signs. The indenting drove me mad sometimes.
- thanks for the static typing and the compilation. I found that the hardest part was to get the program to compile.
Once the program compiled all right it usually worked as intended.
- I know that pattern matching and guards are only syntaxic sugar for case statements, but I found them (the patterns, the guards) a lot more elegant and easy to understand
- I need to learn how to deal better with "newtype" and pattern matching. I used "newtype" instead of type for some data types because I needed to declare them as instances of Show and Read, but that forced me to use as-patterns for game state.
Surely there's simpler ways to do that
- yes, I know about $ to avoid parenthesis for function parameters, but with a good editor it's ok to manage

Now, if anybody has tips on how to improve the code I'll take them on board. I'm going to look at separating the data from the code, maybe with a simple DSL and parser. And yes, the state or IORef monads...