instance Data Map where -- half done!

While testing my JSON serializer/deserializer, I made the (unpleasant) discovery that some standard structures (most notable Data.Map.Map) do not implement all of Data.Generics.Data interface. They just call error on things like toConstr and gunfold. Which is not good, since I use these methods. Why they can't just pretend that a Map K v is for generics a map of (k,v) I don't understand (they implement gfoldl that way), but hey, I'm only a humble learner of Haskell...

So I spent a few hours trying to modify my JSON code to work around this issue, to no avail (well I can get the serialization all right, but the deserialization seems to be tricky). So I've got another approach to work: since the default for Data instances is not right when your objects contains Map structures, lets rewrite these... Using the source for Data.Map and the documentation for Data.Generics.Data, I managed to get it to work, but of course the price is loads (loads, loads) of boilerplate code.

So here goes: I have the simple object: (assume import qualified Data.Map as M)

data MapObjF=MapObjF (M.Map String Int) String
 deriving (Eq,Typeable,Show)

So MapObjF contains a Map String Integer and a String

And here's the Data instance definition (big breath):

instance Data MapObjF where
     gfoldl k z (MapObjF m s) = k (k (z (MapObjF . M.fromList)) (M.toList m)) s
     gunfold k z _ = k (k (z (MapObjF . M.fromList)))
     toConstr (MapObjF _ _) = con_MapObjF
     dataTypeOf _ = ty_MapObjF

con_MapObjF = mkConstr ty_MapObjF "MapObjF" [] Prefix
ty_MapObjF   = mkDataType "MyModule.MapObjF" [con_MapObjF]

I define the constructor, the datatype, and implement gfoldl and gunfold transforming the map into a list of tuples. Of course if you have several constructors, loads of fields it soon becomes unwieldy. Now, is there a Haskell macro system so I can easily generate all that boilerplate for all my data types? Noooo I don't want to leave Haskell for LISP...

Deserializing JSON to Haskell Data objects

Since last week I found a few other posts and libraries on the subject of JSON serialization and deserialization: here and here for example. Nonetheless I've continued on my own path, since the best way of learning is doing. It took me a while to figure out how to reconstruct a Haskell object, and I've only got some limited functionnality to work, but it works. I looked into the source code for gread for clues, I have to say.

So what we want is simple enough:

jsonToObj :: forall a. Data a => JSON.Value -> a

Given a JSON value we want an haskell object, and hopefully the type will match... OK, I'm a bit terse on the error handling side of things at the moment!
(Note: for the code to work you need to import: Control.Monad.State, Data.Generics, Data.List, qualified Data.Map as M, Data.Maybe)

It's simple enough for Strings and Bools:

jsonToObj (JSON.String s)=fromJust $ cast s
jsonToObj (JSON.Bool b)=fromJust $ cast b

Of course, if you expected a Foo and parse a JSON.String, this will fail(The cast return a Maybe). You have to cast since the signature doesn't force Strings or Bools, only Data.

For other types of objects (including numbers, because cast doesn't perform number conversion I found) it's a bit more complicated. Basically we have to figure out the type we want, find a constructor to build an instance of that type, and pass to that constructors proper values from the JSON object.
To find the type we want, I use funny code find in gread:

myDataType = dataTypeOf (getArg getType)
getArg :: a' -> a'
getArg = undefined
getType :: a
getType = undefined

This create a dummy function returning my type, and a dummy function returning what it gets as parameter. They don't need to be implemented, the compiler only cares about the type.

The meat of the function, that decides what constructor to invoke and what data to useas constructor parameter is a bit more involved:

(values,cons)=case x of
 JSON.Object m -> let
  c=indexConstr myDataType 1
  in ((map (\x->fromJust $ M.lookup x m) (constrFields c)),c)
 JSON.Number fl -> ([],if isPrefixOf "Prelude.Int" (dataTypeName myDataType)
  then mkIntConstr myDataType (round fl)
  else mkFloatConstr myDataType fl)
 JSON.Array [] -> ([],indexConstr myDataType 1)
 JSON.Array (x:xs) -> ([x,(JSON.Array xs)],indexConstr myDataType 2)

This snippet calculates the objects to iterator over and the constructor index to use. There's a bit a hand waving there: for objects we will take the first construtor we defined (I'll implement multiple constructor support later), and the JSON values it contains in the map. We just ensure we get the values in the order the fields are defined (given by the constrFields function). For number we use the int constructor if our result type looks like an int, otherwise we use the float constructor. There is probably a better way to construct a number from a Double, but I still need to find it. For the moment we look if the type name starts with "Prelude.Int", which is arguably not very Haskelly. For Arrays, we need to recreate the (head,rest) tuple that gave me trouble when serializing, so we deal first with the head of the list, and put the rest afterwards. The empty list is the first constructor, a non empty the second.

Then to actually create the object we pass the values inside a State monad:

State f=(fromConstrM (State (\(x:xs) -> (jsonToObj x,xs))) cons)

For the list of values we convert the head from json and keep the rest as state. This simple line was the result of intense thinking, I implemented my own State monad first to really understand what I needed to do and then figured out that the State monad did the exact same thing. The final working code is always shorter that all the previous failed attempts!
We then only need to call the State function with the values we calculated earlier, and that give us our full code:

jsonToObj x=
 fst $ f values
 where
  getArg :: a' -> a'
  getArg = undefined
  getType :: a
  getType = undefined
  myDataType = dataTypeOf (getArg getType)
  (values,cons)=case x of
   JSON.Object m -> let
    c=indexConstr myDataType 1
    in ((map (\x->fromJust $ M.lookup x m) (constrFields c)),c)
   JSON.Number fl -> ([],if isPrefixOf "Prelude.Int" (dataTypeName myDataType)
    then mkIntConstr myDataType (round fl)
    else mkFloatConstr myDataType fl)
   JSON.Array [] -> ([],indexConstr myDataType 1)
   JSON.Array (x:xs) -> ([x,(JSON.Array xs)],indexConstr myDataType 2)
  State f=(fromConstrM (State (\(x:xs) -> (jsonToObj x,xs))) cons)

Serializing Haskell objects to JSON

I'm trying to do some simple serialization of Haskell objects to save some state to disk. After tearing some of my hair out debugging parse errors due to silly code I'd written in Read instances declarations, I've decided to use another approach. I'm going to save objects as JSON, since I've already used that JSON library.
The first task is to write generic code that can serialize a Data instance to JSON. This has probably been done somewhere else, but I need to learn, right? So I took a deep breath and dived into Data.Generics.
I quickly rounded up the issues I would face. Most notably, Strings and lists are accessed without any syntaxic sugar, so to speak: Strings are lists of Char, and lists are made of two fields, the head and the rest. Of course, you say, but from that I need to regenerate String and lists of JSON Value objects.
So, to start, how to recognize Strings to treat them differently than other algrebraic types:

isString :: Data a => a -> Bool
isString a = (typeOf a) == (typeOf "")

There's probably better ways to do that, just tell me (-:

Lists (that are not strings) can be recognized with abstract representations of constructors, which are equals regardless of the actual type of elements in the list

isList :: Data a => a -> Bool
isList a
   | isString a = False
   | otherwise  = (typeRepTyCon (typeOf a)) == (typeRepTyCon $ typeOf ([]::[Int]))

Now, transforming a list of the form (head,rest) to [head1,head2...]

jsonList :: Data a => a -> [JSON.Value]
jsonList l=
   concat $ gmapQ f l
   where f b
       | (isList b)= jsonList b
       | otherwise = [objToJson b]

For each element (the actual number depends on whether we're the empty list or not) we either reapply the same method, if it's the inner list or we simply transform to JSON

And then the actual method on objects:

objToJson :: Data a => a -> JSON.Value
objToJson o | isString o=JSON.String (fromJust $ ((cast o)::(Maybe String)))
objToJson o | isList o=JSON.Array (jsonList o)
objToJson o | otherwise=
   let
       c=toConstr o
   in
       case (constrRep c) of
           AlgConstr _-> JSON.Object (Data.Map.fromList(zip (constrFields c) (gmapQ objToJson o)))
           StringConstr s -> JSON.String s
           FloatConstr f -> JSON.Number f
           IntConstr i -> JSON.Number (fromIntegral i)

We first handle Strings, then list, then general objects using constrRep to distinguish between algebraic types that create JSON objects with the proper field names (using constrFields) and other types for JSON primitives.

And that's it for the serialization! The Generics package is not that hard to use but you have to look up examples to figure out how actually use the functions like gmapQ and such...

Now, I have to work on the opposite process: given a type and JSON data, reconstruct the objects... More Haskell fun!

Scraping my boilerplate: Generics instead of Records

I was talking last week about my trouble writing simple code to access and update record field in Haskell. While talking with justin about it, he suggested using parameter type for record object, so that the compiler would check that an update method would actually take as parameters the proper rating and the corresponding update function. That was cool enough, and that got me into the "Scrap your boilerplate" papers and Data.Generics modules. What I have now, using generics programming, is fairly simple and I think I'm getting closer to what Haskell code should be like.
First, Ratings: a Rating has three int values: the current level, normal level, and the experience points. What I do now is that each value is actually typed, and the Rating only holds a list of them:

data RatingScoreType=Normal |
    Current |
    Experience
    deriving (Show,Read,Enum,Eq,Typeable,Data)

data RatingScore=RatingScore RatingScoreType Int
    deriving (Show,Read,Typeable,Data)
data Rating=Rating [RatingScore]
    deriving (Typeable,Data)
  

With that I can add an update method that only changes a RatingScore if the type match with the type provided:

addRS :: RatingScoreType -> Int -> RatingScore -> RatingScore
addRS  t2 b rs@(RatingScore t1 a)
    | t1==t2=RatingScore t1 (a+b)
    | otherwise=rs

And a method that tells me if a score match the given type

getRS :: RatingScoreType -> RatingScore -> Bool
getRS t2 rs@(RatingScore t1 a) 
    | t1==t2=True
    | otherwise=False

(note that addRS can be rewritten using getRS but we don't gain anything in code size)

I can then use Data.Generics functions to provide generic update and get functions:

addR :: Data a => RatingScoreType -> Int -> a -> a
addR rst i=everywhere (mkT (addRS rst i))

getR :: Data a => RatingScoreType -> a -> Int
getR rst a=
    let (RatingScore t1 b)= head $ listify (getRS rst) a
    in b

(There are probably other and maybe better way of writing these, but hey, the documentation is not very rich in examples...)

The level above Ratings are Characteristics: a character holds an array of CharacteristicRating:

data CharacteristicRating = CharacteristicRating Characteristic Rating
    deriving (Show,Read,Typeable,Data)

Where Characteristic is again an Enum of all possible characteristics.

I define the similar 4 functions as above, except they filter on Characteristic first and RatingScoreType second:

addC :: Characteristic -> RatingScoreType -> Int -> CharacteristicRating -> CharacteristicRating
addC c2 rst i cr@(CharacteristicRating c1 r) 
    | c1==c2=everywhere (mkT (addRS rst i)) cr
    | otherwise=cr

getC :: Characteristic -> RatingScoreType -> CharacteristicRating -> Bool
getC c2 rst cr@(CharacteristicRating c1 r) 
    | c1==c2=True
    | otherwise=False

addCharacteristic :: Data a => Characteristic -> RatingScoreType -> Int -> a -> a
addCharacteristic c rst i a = everywhere (mkT (addC c rst i)) a

getCharacteristic :: Data a => Characteristic -> RatingScoreType -> a -> Int
getCharacteristic c rst a = 
    let cr=head $ listify (getC c rst) a
    in getR rst cr

This is pretty cool: if I have a Character c:

getCharacteristic Strength Current c

Gives me the current strength! And a function can take a Data instance (a Character or anything else) and a Characteristic and do both testing and changing value in perfect safety!

Now I'm only just starting playing the Data.Generics, and more generally with Haskell program design, but this is cool. My RPG is not progressing fast, but the main game here is actually building it, not playing it!!

Java and higher order generics

Funny coincidence: yesterday in work I was trying to find a way to express something simple enough in Java: I want a method map that takes a generic collection and return a collection of the same type (that is, passing a List should return a List, passing a Set should return a Set) with a different generic type. In Java you can have:

public <T1,T2> Collection<T2> map(Collection<T1> c,Mapper<T1,T2>

Where Mapper is an interface defining the method T2 map(T1 o);
There is no way I think in Java to express that if you pass a subclass of collection you get the same subclass back.

I tried things like <T1,T2,C extends Collection,C1 extends C<T1>,C2 extends C<T2> and other crazy things that my compiler didn't appreciate.


You have to duplicate methods, and that's what we've done in work:

public <T1,T2> List<T2> map(List<T1> c,Mapper<T1,T2>
public <T1,T2> Set<T2> map(Set<T1> c,Mapper<T1,T2>

And of course a colleague needed a Vector because he was using an old library still using Vectors...
And then today I found that blog post, that point to that PDF document. Apparently you can do this kind of things in Scala and Haskell... Do I need to convince my company to move to Scala?

VList in Java

Two days ago I was reading this article about LISP like lists implemented in Java,
which reminded me I had written an implementation of a VList in Java. It can be found
here, along with a unit test.I make no garantee this is the best or fastest implementation possible!

It implements: get(n), first (head in Haskell), rest() (tail in Haskell), size(), iterator() and map(Mapper), where Mapper is the mapping interface I mentioned earlier.

It's using Java Generics of course. Am I right in thinking there is no way with Java generics to get rid of the compilation warnings and of the necessity of passing the Class object in the code that creates a new array of the generic type?

Java properties with no language changes (oh no!)

There's been a lot of discussions on the web about Java and the support for bean properties. People would like to have a mechanism that relies less on java bean conventions (reflection on method names, find getters and setters) and more on explicit language constructs. The goal is to simplify bean and property handling, have stronger type checks, be able to reference the property and not its values, etc... Most proposals involve additions to the Java syntax. I don't want to add to the confusion, but I just played around with what we have now in Java 5.
What we could have is instead of defining classes directly with fields in them, we could have them reference Property objects instead. Property objects carry the type information and the value itself. An inheritance hierarchy can provide for read-only, write-only and read-write hierarchy. Using public final fields we can have an access syntax that is not the same as the current getter and setter methods, but pretty close.
Example:

public class PersonBean {
    public final IFullProperty name=new NonNullProperty();
   
    public final IReadProperty id;
   
    public final IFullProperty email=new FullProperty(){
        @Override
        public void set(String value) {
            if (value.indexOf('@')==-1){
                throw new IllegalArgumentException("email does not contain @");
            }
            super.set(value);
        }
    };
   
    public PersonBean(int id,String name){
        this.name.set(name);
        this.id=new ReadOnlyProperty(id);
    }
}

Of course, the generic madness means the type information has to be typed twice, but when generics syntax is cleaned up it'd be a lot more readable. This bean says that it has a changeable name property, a String that cannot be null. It has an id integer property that cannot be changed. Finally it has an email property that has ad hoc constraint checking. The Property interfaces and its sub interfaces define a get() and a set() method so accessing the fields becomes:

bean.getName() -> bean.name.get()
bean.setName(name) -> bean.name.set(name)

The property change listener can be attached to the property object so that the bean doesn't have to have any line of code related to listeners. Everything can be done via static methods:

static void addPropertyChangeListener(Object bean,PropertyChangeListener pcl);

to add a listener on all properties

static void addPropertyChangeListener(Object bean,PropertyChangeListener pcl,IProperty p);

to add a listener on a specific property. Note that since we're taking a property object, and not a property name for example we cannot register a listener on a non existing property. The property does not have a back link to its bean (to avoid disgracious constructors with this), so we need to have both the bean and the propert.
Example:
addPropertyChangeListener(bean,listener,bean.email)

If anybody's interested, all the code with unit tests is available here.