As I've explained in several places, we have an issue in BuildWrapper (the underlying Haskell program for the EclipseFP plugins): BuildWrapper depends on both the GHC API and the Cabal API. Since GHC itself has a dependency on Cabal, once you've installed GHC, BuildWrapper can only use the same version of the Cabal API that was used by that GHC. Moreover, some Cabal files (setup-config in the dist directory for example) are cabal version dependent (since they only serialize Haskell structures, this is for safety). The net result is that if you install a newer version of Cabal and cabal-install, BuildWrapper still uses the old version, and hence the Cabal library BuildWrapper uses and the cabal-install executable have different versions, and hell breaks loose.
I have looked at solving the issue in GHC itself: GHC does not need to have a dependency to the Cabal API. I had started investigating with help from GHC devs, but it was end of October, and I had understood that GHC 7.8 was to be released in November, so there was no time to do such a change. So I thought I would wait for the next release. But it's now end of December, and GHC 7.8 still hasn't been released. So should we have to wait another year to solve that issue? Gasp!
But then, out of the blue, help came! Benno Fünfstück released dynamic-cabal, with the promise that it would solve exactly this issue! I actually should have had that idea myself, but hey... Basically, dynamic-cabal provides simpler structures than the full Cabal API, and extracts them from the Cabal API via dynamic code generation. The code is generated and ran dynamically using the GHC API. Since the running code itself doesn't depend on GHC, it can use the latest version of the Cabal library, and hence read the setup-config file properly!
I have now rewritten parts of BuildWrapper to use dynamic-cabal, and I only had to add minor enhancements to dynamic-cabal. I have upgraded my own cabal-install to 1.18 and everything works (my GHC still uses Cabal 1.16)!! I'm not releasing that version yet because it will provide some new features for EclipseFP 2.6, but if really you want to give it a try just build it from Github.
So my thanks go to Benno! A big hurdle has been cleared on EclipseFP's path to world domination!
Showing posts with label GHC. Show all posts
Showing posts with label GHC. Show all posts
Monday, December 30, 2013
Friday, October 07, 2011
New backend for EclipseFP
A while ago I was wondering how I could enhance EclipseFP, regarding things like memory usage, support for multi components projects, etc. I now have a working version for adventurous testers!
The main thing is the removal of Scion. EclipseFP now uses another backend component called "buildwrapper". There is no server, buildwrapper is only an executable that can be launched with flags, and writes a JSON result on the standard output. This means no more huge memory usage, and less synchronization headaches.
Buildwrapper uses a copy of the project inside a hidden folder. This means that when we want to get say build errors on a dirty editor content, we can write the dirty content to the file inside the hidden folder, and launch a build process on these files, thus avoiding the limitations that the GHC API has on building from a StringBuffer (doesn't work for files requiring preprocessing, etc).
We only use the GHC API to get the types of things in a source file, but we use Cabal to build the project. Buildwrapper actually launches the cabal executable and parses the results. I know, sounds crude compare to an API only approach, but I have the feeling it just works better. Basically the IDE will now behave a lot close to what you get if you were building your project from the command line. And then you get additional goodies, like multi component support (so you can have the executable of your project reference the library in the same project, and it works!).
Buildwrapper uses haskell-src-exts for outline. I've actually seen cases where it needed language pragmas to parse correctly that GHC didn't need, but such is life.
Performance seems to be quite good, except for things that use the GHC API, where there's a little lag. Hopefully the fact that we don't use gigs of memory any more will make up for it.
I'm also thinking about distribution. EclipseFP came with Scion and Scion-browser bundled, if only because we used a different version of scion than the one on Hackage. I think that moving forward we'll rely on cabal install and Haskage. I'm going to add a simple interface to run cabal install inside EclipseFP, so you shouldn't have to go outside to install a package. So the EclipseFP preferences will just ask the user for the path to the buildwrapper and scion-browser executables, with an option to install them from hackage.
If people want to start testing, just go on github, and grab the BuildWrapper repository (main branch) and the EclipseFP project (buildwrapper branch). For the moment, BuildWrapper probably only builds on GHC 7.0. Once I have the feeling it's robust enough and there's no big flaw in the new architecture, I can work on enhancing portability.
The main thing is the removal of Scion. EclipseFP now uses another backend component called "buildwrapper". There is no server, buildwrapper is only an executable that can be launched with flags, and writes a JSON result on the standard output. This means no more huge memory usage, and less synchronization headaches.
Buildwrapper uses a copy of the project inside a hidden folder. This means that when we want to get say build errors on a dirty editor content, we can write the dirty content to the file inside the hidden folder, and launch a build process on these files, thus avoiding the limitations that the GHC API has on building from a StringBuffer (doesn't work for files requiring preprocessing, etc).
We only use the GHC API to get the types of things in a source file, but we use Cabal to build the project. Buildwrapper actually launches the cabal executable and parses the results. I know, sounds crude compare to an API only approach, but I have the feeling it just works better. Basically the IDE will now behave a lot close to what you get if you were building your project from the command line. And then you get additional goodies, like multi component support (so you can have the executable of your project reference the library in the same project, and it works!).
Buildwrapper uses haskell-src-exts for outline. I've actually seen cases where it needed language pragmas to parse correctly that GHC didn't need, but such is life.
Performance seems to be quite good, except for things that use the GHC API, where there's a little lag. Hopefully the fact that we don't use gigs of memory any more will make up for it.
I'm also thinking about distribution. EclipseFP came with Scion and Scion-browser bundled, if only because we used a different version of scion than the one on Hackage. I think that moving forward we'll rely on cabal install and Haskage. I'm going to add a simple interface to run cabal install inside EclipseFP, so you shouldn't have to go outside to install a package. So the EclipseFP preferences will just ask the user for the path to the buildwrapper and scion-browser executables, with an option to install them from hackage.
If people want to start testing, just go on github, and grab the BuildWrapper repository (main branch) and the EclipseFP project (buildwrapper branch). For the moment, BuildWrapper probably only builds on GHC 7.0. Once I have the feeling it's robust enough and there's no big flaw in the new architecture, I can work on enhancing portability.
Friday, June 24, 2011
EclipseFP: the way forward??
This is not the usual "hey I managed to get some code to work, check it out" blog post. Bear with me as I'm just thinking out loud. This is about EclipseFP and how we can move it forwards.
At the moment EclipseFP works pretty well for simple, smallish Haskell projects. People have noted some performance issues on large projects, and part of it is the huge memory footprint of GHC when used through its API. After a couple of hours of working on a yesod project (so we're talking quasi quotation, template haskell, etc), the scion server takes 1Gb of memory... Restarting brings it back to reasonable levels, which seems to point to memory leaks in the GHC API.
Maybe more critically, people are now asking for features that get increasingly harder to implement using Scion:
- static GHC flags like -threaded (Nominolo's working on a version of Scion that could handle these)
- C or HSC sources (or other kind of stuff requiring a preprocessor)
- Having executables or test suites referencing the library component directly in the Cabal file
The main thing with these is that Cabal handles them fine, so basically typing cabal build at the command prompt solves these issues, which undermines the usefulness of having an IDE in the first place... But they require more than just calling the GHC API: building and linking a library in-place, calling a c compiler, hsc2hs, etc...
Of course Cabal publishes an API. So for example the code used by Cabal to build an in-place version of the library I could access, in theory, from Scion. But then Cabal calls the ghc executable, so the Cabal API gives me no easy way to get back errors from GHC, etc...
What does accessing the GHC API gives us? An AST, that can be generated from an unsaved file (so you don't need to save your haskell source file to see your errors, our up to date outline, etc). All the rest (syntax highlighting, jump to definition, etc) could probably be done from the AST.
So I'm wondering now if scrapping scion totally would not be a good idea. A rough outline of how things could work would be:
- create a temp folder in which we will work, copy in it all the project files (sources, etc)
- use cabal to build in that folder
- parse cabal output to get errors and their location
- unsaved files in the EclipseFP editor could have their unsaved contents copied into that folder (that folder represent the current state of the project, which may not be what is really saved on disk)
- have only a simple Haskell executable that given a module file, loads it using all the built files in the temp folder, and return the AST in some easy to parse format. Firing and getting the result would ensure GHC cannot grab loads of memory and not release it
- the IDE code then gets the AST and does all it needs to it
- the AST could even be saved in that temp folder in some format, so that the Java code would only request it when the original file is more recent, which would allow easy parsing of dependent modules AST (for example to retrieve all the symbols exported, etc)
There are a lot of ugly bits in there, maybe, but that would solve my problems... I suppose all of that could be written in Haskell and we could keep a similar enough API to scion, so that the Java code wouldn't need change too much...
For the moment I'm not going to reach into things anyway, but if anybody has some feedback, I'll gladly take any opinion on board!
At the moment EclipseFP works pretty well for simple, smallish Haskell projects. People have noted some performance issues on large projects, and part of it is the huge memory footprint of GHC when used through its API. After a couple of hours of working on a yesod project (so we're talking quasi quotation, template haskell, etc), the scion server takes 1Gb of memory... Restarting brings it back to reasonable levels, which seems to point to memory leaks in the GHC API.
Maybe more critically, people are now asking for features that get increasingly harder to implement using Scion:
- static GHC flags like -threaded (Nominolo's working on a version of Scion that could handle these)
- C or HSC sources (or other kind of stuff requiring a preprocessor)
- Having executables or test suites referencing the library component directly in the Cabal file
The main thing with these is that Cabal handles them fine, so basically typing cabal build at the command prompt solves these issues, which undermines the usefulness of having an IDE in the first place... But they require more than just calling the GHC API: building and linking a library in-place, calling a c compiler, hsc2hs, etc...
Of course Cabal publishes an API. So for example the code used by Cabal to build an in-place version of the library I could access, in theory, from Scion. But then Cabal calls the ghc executable, so the Cabal API gives me no easy way to get back errors from GHC, etc...
What does accessing the GHC API gives us? An AST, that can be generated from an unsaved file (so you don't need to save your haskell source file to see your errors, our up to date outline, etc). All the rest (syntax highlighting, jump to definition, etc) could probably be done from the AST.
So I'm wondering now if scrapping scion totally would not be a good idea. A rough outline of how things could work would be:
- create a temp folder in which we will work, copy in it all the project files (sources, etc)
- use cabal to build in that folder
- parse cabal output to get errors and their location
- unsaved files in the EclipseFP editor could have their unsaved contents copied into that folder (that folder represent the current state of the project, which may not be what is really saved on disk)
- have only a simple Haskell executable that given a module file, loads it using all the built files in the temp folder, and return the AST in some easy to parse format. Firing and getting the result would ensure GHC cannot grab loads of memory and not release it
- the IDE code then gets the AST and does all it needs to it
- the AST could even be saved in that temp folder in some format, so that the Java code would only request it when the original file is more recent, which would allow easy parsing of dependent modules AST (for example to retrieve all the symbols exported, etc)
There are a lot of ugly bits in there, maybe, but that would solve my problems... I suppose all of that could be written in Haskell and we could keep a similar enough API to scion, so that the Java code wouldn't need change too much...
For the moment I'm not going to reach into things anyway, but if anybody has some feedback, I'll gladly take any opinion on board!
Monday, September 28, 2009
Haskell development in Eclipse
I have started contributing to the eclipsefp project. For those who don't know, it's a Eclipse-based environment for developing in Haskell.
EclipseFP has had a troubled history so far. Leif started the project and developed an impressive set of features. He then pursued the Cohatoe project to be able to write Eclipse plugins inm Haskell, in a effort to avoid rewriting in Java all kind of things that already existed in Haskell. And then this summer Thomas started integrating with Scion, which is a Haskell library that communicates with JSON data over simple sockets with non Haskell clients. Scion offers method to load Cabal projects, type check files, etc... That work is not finished, but Leif and Thomas have done such a great job that it's easy enough now to finish the missing bits here and there.
This is perfect for me: part of the work is in Java and Eclipse, which is a lot of what I do in my day job, so I can bring a lot of experience there. Scion is a small enough library that need to be extended to provide me with all I want, so I can do little enhancements in Haskell and slowly build on features. And, last but not least, I need to learn the GHC API (which is what Scion uses to parse, check and compile files), to find what I need for.
This will only enhance my knowledge of Haskell, and I hope that I'll be able to make something useful to provide Haskell hackers with another IDE to develop in.
For those interested, the github repositories where my current work can be found are:
git://github.com/JPMoresmau/scion.git
git://github.com/JPMoresmau/eclipsefp.git
EclipseFP has had a troubled history so far. Leif started the project and developed an impressive set of features. He then pursued the Cohatoe project to be able to write Eclipse plugins inm Haskell, in a effort to avoid rewriting in Java all kind of things that already existed in Haskell. And then this summer Thomas started integrating with Scion, which is a Haskell library that communicates with JSON data over simple sockets with non Haskell clients. Scion offers method to load Cabal projects, type check files, etc... That work is not finished, but Leif and Thomas have done such a great job that it's easy enough now to finish the missing bits here and there.
This is perfect for me: part of the work is in Java and Eclipse, which is a lot of what I do in my day job, so I can bring a lot of experience there. Scion is a small enough library that need to be extended to provide me with all I want, so I can do little enhancements in Haskell and slowly build on features. And, last but not least, I need to learn the GHC API (which is what Scion uses to parse, check and compile files), to find what I need for.
This will only enhance my knowledge of Haskell, and I hope that I'll be able to make something useful to provide Haskell hackers with another IDE to develop in.
For those interested, the github repositories where my current work can be found are:
git://github.com/JPMoresmau/scion.git
git://github.com/JPMoresmau/eclipsefp.git
Subscribe to:
Posts (Atom)