Scratching an itch: generating Cabal data-files field automatically

Maybe I didn't look hard enough, but I'm not aware of a tool to generate the contents of the Cabal data-files field automatically when you have loads of folders to include. Cabal has very simple wildcard matching for files references in this field, by design (to avoid including too much data in a source distribution). So it only supports wildcards to replace the file name inside a directory for a given extension, and doesn't support sub directories.

For the reload project - first release on Hackage! - I had to include loads of files, all the Polymer web components the UI depends on, which are all on different sub directories, with a bunch of different extensions. So I wrote a little tool to generate the field automatically, and put it on Hackage too.

You pass it a directory name, possibly some subdirectories and extensions to ignore, and it generates all the required entries. Saved me loads of time, and scratched my own itch!

Dynamic-cabal to solve dependency hell in BuildWrapper

As I've explained in several places, we have an issue in BuildWrapper (the underlying Haskell program for the EclipseFP plugins): BuildWrapper depends on both the GHC API and the Cabal API. Since GHC itself has a dependency on Cabal, once you've installed GHC, BuildWrapper can only use the same version of the Cabal API that was used by that GHC. Moreover, some Cabal files (setup-config in the dist directory for example) are cabal version dependent (since they only serialize Haskell structures, this is for safety). The net result is that if you install a newer version of Cabal and cabal-install, BuildWrapper still uses the old version, and hence the Cabal library BuildWrapper uses and the cabal-install executable have different versions, and hell breaks loose.

I have looked at solving the issue in GHC itself: GHC does not need to have a dependency to the Cabal API. I had started investigating with help from GHC devs, but it was end of October, and I had understood that GHC 7.8 was to be released in November, so there was no time to do such a change. So I thought I would wait for the next release. But it's now end of December, and GHC 7.8 still hasn't been released. So should we have to wait another year to solve that issue? Gasp!

But then, out of the blue, help came! Benno Fünfstück released dynamic-cabal, with the promise that it would solve exactly this issue! I actually should have had that idea myself, but hey... Basically, dynamic-cabal provides simpler structures than the full Cabal API, and extracts them from the Cabal API via dynamic code generation. The code is generated and ran dynamically using the GHC API. Since the running code itself doesn't depend on GHC, it can use the latest version of the Cabal library, and hence read the setup-config file properly!

I have now rewritten parts of BuildWrapper to use dynamic-cabal, and I only had to add minor enhancements to dynamic-cabal. I have upgraded my own cabal-install to 1.18 and everything works (my GHC still uses Cabal 1.16)!! I'm not releasing that version yet because it will provide some new features for EclipseFP 2.6, but if really you want to give it a try just build it from Github.

So my thanks go to Benno! A big hurdle has been cleared on EclipseFP's path to world domination!

Manage Cabal packages from inside EclipseFP

I'm thinking the new version of EclipseFP will not bundle the Haskell code we use, but will download it from Hackage, which would make sense: we would be able to update the Haskell side without requiring a EclipseFP upgrade, and vice versa. I would expect Haskell developers would be quite used to install packages from Hackage using cabal install. However, let's try to minimize the number of things that a user would need to go outside of EclipseFP to do on a normal Haskell project. So I've added another view:

This view lets you see the list of packages you've installed, with their version, or the full list from Hackage, that you can filter by typing the prefix of the name you're looking for. From there you can install the packages as either Global or User, see the package information, or click on the "More Information" link to open the Hackage page directly.
You can also update the list. This GUI is only a wrapper above the cabal list and cabal info commands.
Hopefully this will be useful to people! Should I add a text field somewhere to be able to specify more options to cabal install, or is that overkill for a GUI?

New backend for EclipseFP

A while ago I was wondering how I could enhance EclipseFP, regarding things like memory usage, support for multi components projects, etc. I now have a working version for adventurous testers!

The main thing is the removal of Scion. EclipseFP now uses another backend component called "buildwrapper". There is no server, buildwrapper is only an executable that can be launched with flags, and writes a JSON result on the standard output. This means no more huge memory usage, and less synchronization headaches.

Buildwrapper uses a copy of the project inside a hidden folder. This means that when we want to get say build errors on a dirty editor content, we can write the dirty content to the file inside the hidden folder, and launch a build process on these files, thus avoiding the limitations that the GHC API has on building from a StringBuffer (doesn't work for files requiring preprocessing, etc).

We only use the GHC API to get the types of things in a source file, but we use Cabal to build the project. Buildwrapper actually launches the cabal executable and parses the results. I know, sounds crude compare to an API only approach, but I have the feeling it just works better. Basically the IDE will now behave a lot close to what you get if you were building your project from the command line. And then you get additional goodies, like multi component support (so you can have the executable of your project reference the library in the same project, and it works!).

Buildwrapper uses haskell-src-exts for outline. I've actually seen cases where it needed language pragmas to parse correctly that GHC didn't need, but such is life.

Performance seems to be quite good, except for things that use the GHC API, where there's a little lag. Hopefully the fact that we don't use gigs of memory any more will make up for it.
I'm also thinking about distribution. EclipseFP came with Scion and Scion-browser bundled, if only because we used a different version of scion than the one on Hackage. I think that moving forward we'll rely on cabal install and Haskage. I'm going to add a simple interface to run cabal install inside EclipseFP, so you shouldn't have to go outside to install a package. So the EclipseFP preferences will just ask the user for the path to the buildwrapper and scion-browser executables, with an option to install them from hackage.

If people want to start testing, just go on github, and grab the BuildWrapper repository (main branch) and the EclipseFP project (buildwrapper branch). For the moment, BuildWrapper probably only builds on GHC 7.0. Once I have the feeling it's robust enough and there's no big flaw in the new architecture, I can work on enhancing portability.

EclipseFP: the way forward??

This is not the usual "hey I managed to get some code to work, check it out" blog post. Bear with me as I'm just thinking out loud. This is about EclipseFP and how we can move it forwards.

At the moment EclipseFP works pretty well for simple, smallish Haskell projects. People have noted some performance issues on large projects, and part of it is the huge memory footprint of GHC when used through its API. After a couple of hours of working on a yesod project (so we're talking quasi quotation, template haskell, etc), the scion server takes 1Gb of memory... Restarting brings it back to reasonable levels, which seems to point to memory leaks in the GHC API.
Maybe more critically, people are now asking for features that get increasingly harder to implement using Scion:
- static GHC flags like -threaded (Nominolo's working on a version of Scion that could handle these)
- C or HSC sources (or other kind of stuff requiring a preprocessor)
- Having executables or test suites referencing the library component directly in the Cabal file

The main thing with these is that Cabal handles them fine, so basically typing cabal build at the command prompt solves these issues, which undermines the usefulness of having an IDE in the first place... But they require more than just calling the GHC API: building and linking a library in-place, calling a c compiler, hsc2hs, etc...
Of course Cabal publishes an API. So for example the code used by Cabal to build an in-place version of the library I could access, in theory, from Scion. But then Cabal calls the ghc executable, so the Cabal API gives me no easy way to get back errors from GHC, etc...

What does accessing the GHC API gives us? An AST, that can be generated from an unsaved file (so you don't need to save your haskell source file to see your errors, our up to date outline, etc). All the rest (syntax highlighting, jump to definition, etc) could probably be done from the AST.

So I'm wondering now if scrapping scion totally would not be a good idea. A rough outline of how things could work would be:
- create a temp folder in which we will work, copy in it all the project files (sources, etc)
- use cabal to build in that folder
- parse cabal output to get errors and their location
- unsaved files in the EclipseFP editor could have their unsaved contents copied into that folder (that folder represent the current state of the project, which may not be what is really saved on disk)
- have only a simple Haskell executable that given a module file, loads it using all the built files in the temp folder, and return the AST in some easy to parse format. Firing and getting the result would ensure GHC cannot grab loads of memory and not release it
- the IDE code then gets the AST and does all it needs to it
- the AST could even be saved in that temp folder in some format, so that the Java code would only request it when the original file is more recent, which would allow easy parsing of dependent modules AST (for example to retrieve all the symbols exported, etc)

There are a lot of ugly bits in there, maybe, but that would solve my problems... I suppose all of that could be written in Haskell and we could keep a similar enough API to scion, so that the Java code wouldn't need change too much...

For the moment I'm not going to reach into things anyway, but if anybody has some feedback, I'll gladly take any opinion on board!