tag:blogger.com,1999:blog-374042882024-03-07T07:56:36.794+01:00JP Moresmau's Programming BlogIn this blog I talk about some of the personal programming I do as a hobby. From Java to Rust via Haskell, I've played around with a lot of technologies and still try to have fun with new languages and APIs!JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.comBlogger170125tag:blogger.com,1999:blog-37404288.post-7000955208690813062023-06-16T14:15:00.000+02:002023-06-16T14:15:14.651+02:00Experimenting with car physics in Bevy<p> I was curious about how a racing game, say, would implement the physics of a car (accelerating, braking) in a ECS setting. I found this page <a href="https://asawicki.info/Mirror/Car%20Physics%20for%20Games/Car%20Physics%20for%20Games.html">https://asawicki.info/Mirror/Car%20Physics%20for%20Games/Car%20Physics%20for%20Games.html</a> that described some equations with actual values for an existing car, so it was perfect! I coded the equations as a Bevy system and used some Bevy UI to be able to show the speed, RPM, etc. </p><p>So here's a very exciting car racing game, where you can accelerate and brake and see the speed go up and down! <a href="https://github.com/JPMoresmau/carphys">https://github.com/JPMoresmau/carphys</a> for the curious!<br /></p>JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com0tag:blogger.com,1999:blog-37404288.post-2330225732471140242023-06-06T11:59:00.000+02:002023-06-06T11:59:02.883+02:00PRQL to SQL transformation on-the-fly through the Postgres protocol<p>Recently I was researching reverse proxies, and I was wondering how to write such a proxy in <a href="https://github.com/tokio-rs/tokio/blob/master/examples/proxy.rs">Rust</a>. I found a little use case that could be interesting: a reverse proxy that sits in front of PostgreSQL, and understands enough about the <a href="https://www.postgresql.org/docs/current/protocol.html">client/server protocol</a> to intercept queries and do something with them. I discovered <a href="https://github.com/PRQL/prql">PRQL</a>, that claims to be "a modern language for transforming data"and "a simple, powerful, pipelined
SQL replacement". So why not allow on-the-fly PRQL to SQL compilation inside the Postgres protocol?</p><p>So <a href="https://github.com/JPMoresmau/prqlproxy">prqlproxy</a> is a reverse proxy that transmits data between a PostgreSQL client and server, and verifies if when running a query it starts with a magic prefix, in this case "<span style="font-family: courier;">prql:"</span>. In this case it assumes everything after the colon is PRQL and uses the Rust compiler to generate SQL that is then passed on to the database server.</p><p>This means you can write PRQL queries from psql for example (using the <a href="https://github.com/devrimgunduz/pagila">Pagila</a> sample database):</p><div style="background-color: #1f1f1f; color: #cccccc; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #cccccc;">psql -h localhost -p 6142 -d pagila -U postgres</span></div><div><span style="color: #cccccc;">Password for user postgres: </span></div><div><span style="color: #cccccc;">psql (14.7 (Homebrew))</span></div><div><span style="color: #cccccc;">Type "help" for help.</span></div><br /><div><span style="color: #cccccc;">pagila=# prql: from customer</span></div><div><span style="color: #cccccc;">pagila-# select [first_name, last_name]</span></div><div><span style="color: #cccccc;">pagila-# take 1..20;</span></div><div><span style="color: #cccccc;"> first_name | last_name </span></div><div><span style="color: #cccccc;">------------+-----------</span></div><div><span style="color: #cccccc;"> MARY | SMITH</span></div><div><span style="color: #cccccc;"> PATRICIA | JOHNSON</span></div><div><span style="color: #cccccc;"> LINDA | WILLIAMS</span></div><div><span style="color: #cccccc;"> BARBARA | JONES</span></div><div><span style="color: #cccccc;"> ELIZABETH | BROWN</span></div><div><span style="color: #cccccc;"> JENNIFER | DAVIS</span></div><div><span style="color: #cccccc;"> MARIA | MILLER</span></div><div><span style="color: #cccccc;"> SUSAN | WILSON</span></div><div><span style="color: #cccccc;"> MARGARET | MOORE</span></div><div><span style="color: #cccccc;"> DOROTHY | TAYLOR</span></div><div><span style="color: #cccccc;"> LISA | ANDERSON</span></div><div><span style="color: #cccccc;"> NANCY | THOMAS</span></div><div><span style="color: #cccccc;"> KAREN | JACKSON</span></div><div><span style="color: #cccccc;"> BETTY | WHITE</span></div><div><span style="color: #cccccc;"> HELEN | HARRIS</span></div><div><span style="color: #cccccc;"> SANDRA | MARTIN</span></div><div><span style="color: #cccccc;"> DONNA | THOMPSON</span></div><div><span style="color: #cccccc;"> CAROL | GARCIA</span></div><div><span style="color: #cccccc;"> RUTH | MARTINEZ</span></div><div><span style="color: #cccccc;"> SHARON | ROBINSON</span></div><div><span style="color: #cccccc;">(20 rows)</span></div><br /><div><span style="color: #cccccc;">pagila=# prql: I don't know what I'm doing;</span></div><div><span style="color: #cccccc;">ERROR: Unknown name `I`</span></div></div><p></p><p>Note above that PRQL errors are returned to the PostgreSQL client, in this case the server doesn't even get reached.</p><p>Of course, this is a very simple toy program, that only supports the simple query protocol (no binding of variables), but it was fun!<br /></p><p>Code is on <a href="https://github.com/JPMoresmau/prqlproxy">Github</a> of course!<br /></p>JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com0tag:blogger.com,1999:blog-37404288.post-51697664627823815122023-04-21T09:39:00.001+02:002023-04-21T09:39:38.456+02:00Data queries and transformations via WebAssembly plugins<p>Following on the previous research I've done on using WebAssembly as a way to write plugins possibly in several languages and running them in a Rust app via the <a href="http://Wasmer.io">Wasmer.io</a> runtime, I've started building <a href="https://github.com/JPMoresmau/query-plugins">something more extensive</a>.</p><p>The plugins now allow writing data querying and transformation code without having to deal with the low level details on how to connect to the underlying database. A plugin can:</p><p>- define the input parameters it needs to run.</p><p>- based on the actual provided values for these parameters, generate a SQL query string and bound parameters that actually need to run on the database. So a plugin can have full control on the SQL generation. Some plugins could always use the same SQL query and just use bound parameters coming from their input parameters, or could do any kind of preprocessing to generate the SQL, for things that bound parameters don't allow.</p><p>- the runtime will run the query with the bound parameters on the underlying database, and return the result row by row to the plugin. The plugin can then do whatever processing on each row it needs, and return intermediate results or only return results at the end.</p><p>Since I using Wasmer, let's define the <a href="https://github.com/JPMoresmau/query-plugins/blob/main/query-runner/query.wai">interface</a> we need in WAI format:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #6a9955;">// Get the metadata of the query.</span></div><div><span style="color: #dcdcaa;">metadata</span><span style="color: #d4d4d4;">: </span><span style="color: #569cd6;">func</span><span style="color: #d4d4d4;">() -> query-metadata</span></div><br /><div><span style="color: #6a9955;">// Start the query processing with the given variables.</span></div><div><span style="color: #dcdcaa;">start</span><span style="color: #d4d4d4;">: </span><span style="color: #569cd6;">func</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">variables: </span><span style="color: #4ec9b0;">list</span><span style="color: #9cdcfe;"><variable></span><span style="color: #d4d4d4;">) -> execution</span></div><br /><div><span style="color: #6a9955;">// Encapsulates the query row processing.</span></div><div><span style="color: #569cd6;">resource</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">execution</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #6a9955;">// The actual query to run.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">query-string</span><span style="color: #d4d4d4;">: </span><span style="color: #569cd6;">func</span><span style="color: #d4d4d4;">() -> </span><span style="color: #4ec9b0;">string</span></div><br /><div><span style="color: #d4d4d4;"> </span><span style="color: #6a9955;">// The variables to use in the query.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">variables</span><span style="color: #d4d4d4;">: </span><span style="color: #569cd6;">func</span><span style="color: #d4d4d4;">() -> </span><span style="color: #4ec9b0;">list</span><span style="color: #d4d4d4;"><variable></span></div><br /><div><span style="color: #d4d4d4;"> </span><span style="color: #6a9955;">// Callback on each data row, returning potential intermediate results.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">row</span><span style="color: #d4d4d4;">: </span><span style="color: #569cd6;">func</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">data: </span><span style="color: #4ec9b0;">list</span><span style="color: #9cdcfe;"><variable></span><span style="color: #d4d4d4;">) -> </span><span style="color: #4ec9b0;">option</span><span style="color: #d4d4d4;"><query-result></span></div><br /><div><span style="color: #d4d4d4;"> </span><span style="color: #6a9955;">// Callback on query end, returning potential final results.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #6a9955;">// Columns are passed in case no data was returned.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">end</span><span style="color: #d4d4d4;">: </span><span style="color: #569cd6;">func</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">columns: </span><span style="color: #4ec9b0;">list</span><span style="color: #9cdcfe;"><</span><span style="color: #4ec9b0;">string</span><span style="color: #9cdcfe;">></span><span style="color: #d4d4d4;">) -> </span><span style="color: #4ec9b0;">option</span><span style="color: #d4d4d4;"><query-result></span></div><div><span style="color: #d4d4d4;">}</span></div><br /></div><p></p><p>The simple data types are left out for brevity (they are defined in their <a href="https://github.com/JPMoresmau/query-plugins/blob/main/query-runner/query-common.wai">own file</a>), but hopefully the intent of this interface should be clear. The <span style="font-family: courier;">metadata</span> function returns a description of what the plugin does and what parameters it takes (a parameter is strongly typed). The <span style="font-family: courier;">start</span> function take actual values for parameters and return an <span style="font-family: courier;">execution</span> resource. This <span style="font-family: courier;">execution</span> exposes the actual query string and bound parameters for that query (via the <span style="font-family: courier;">variables</span> method). The runtime will then call the <span style="font-family: courier;">row</span> function for each result row and the <span style="font-family: courier;">end</span> function at the end. Each of these can return a possibly partial result. The <span style="font-family: courier;">end</span> function takes the names of columns so that proper metadata is known even if the query returned no row.</p><p>Examples of very basic plugins used in tests can be seen <a href="https://github.com/JPMoresmau/query-plugins/tree/main/test-collect">here</a> and <a href="https://github.com/JPMoresmau/query-plugins/tree/main/test-collect2">here</a>. They just collect the data passed to them and return it in the end method.<br /></p><p>Each plugin can then be compiled to WASM for example via the <span style="font-family: courier;">cargo wapm --dry-run</span> command provided by Wasmer.</p><p>The current runtime I've built is very simple: it takes all plugins from a folder and database connections are defined in a <a href="https://github.com/JPMoresmau/query-plugins/blob/main/query-runner/config/connections.yaml">YAML file</a>, and only Sqlite and Postgres are supported. An <a href="https://github.com/JPMoresmau/query-plugins/tree/main/query-runner-bin">executable</a> is provided to be able to run plugins from the command line.</p><p>Using a WAI interface and not having to deal with low level WASM code is great. <span style="font-family: courier;">cargo expand</span> is your friend to understand what Wasmer generates as structures are generated differently between the <span style="font-family: courier;">import!</span> and <span style="font-family: courier;">export!</span> macros, so some structures own their data while some take references, which can sometimes trip you up.</p><p>Of course I would need to test this for performance, to determine how much copying of data is done between the Rust runtime and the WASM plugins.</p><p>Let me know if you have use cases where this approach could be interesting! As usual, all code is available on <a href="https://github.com/JPMoresmau/query-plugins">Github</a>.<br /></p><p><br /></p>JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com0tag:blogger.com,1999:blog-37404288.post-614539212676190742023-04-11T14:35:00.000+02:002023-04-11T14:35:18.304+02:00WebAssembly: bidirectional communication between components and host<p>Still investigating the use of WebAssembly to implement a plugin system using Rust and <a href="https://wasmer.io/">Wasmer</a>. In the previous post, I could load a WebAssembly plugin that implemented an interface that the host knew about. But since WebAssembly is very limited by design in terms of API, as soon as a plugin wants to interact with the outside world in some shape of form, it needs to be able to call an API the host will provide. Thus the host can ensure safety, performance, etc. without allowing the WebAssembly code direct access to real resources.</p><p>So let's say we'd like to customize our greeting message based on the time of day. We're going to need a very simple function to tell us the current hour of the day:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #dcdcaa;">hour</span><span style="color: #d4d4d4;">: </span><span style="color: #569cd6;">func</span><span style="color: #d4d4d4;">() -> </span><span style="color: #4ec9b0;">u32</span></div></div><p></p><p>This goes into <span style="font-family: courier;">host.wai</span>. In our english-rs module we can then <span style="font-family: courier;">import!</span> this file with standard web assembly Rust generation and use it in our implementation of <span style="font-family: courier;">greet</span>:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #4ec9b0;">wai_bindgen_rust</span><span style="color: #d4d4d4;">::</span><span style="color: #569cd6;">export</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"greeter.wai"</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #4ec9b0;">wai_bindgen_rust</span><span style="color: #d4d4d4;">::</span><span style="color: #569cd6;">import</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"host.wai"</span><span style="color: #d4d4d4;">);</span></div><br /><div><span style="color: #569cd6;">struct</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Greeter</span><span style="color: #d4d4d4;">;</span></div><br /><div><span style="color: #569cd6;">impl</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">crate</span><span style="color: #d4d4d4;">::</span><span style="color: #4ec9b0;">greeter</span><span style="color: #d4d4d4;">::</span><span style="color: #4ec9b0;">Greeter</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">for</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Greeter</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #6a9955;"> </span><span style="color: #6a9955;">/// The language we greet in.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">language</span><span style="color: #d4d4d4;">() </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">from</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"English"</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> }</span></div><br /><div><span style="color: #6a9955;"> </span><span style="color: #6a9955;">/// Greet the given name.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">greet</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">name</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;">) </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">hour</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">host</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">hour</span><span style="color: #d4d4d4;">();</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">if</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">hour</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;"><</span><span style="color: #d4d4d4;"> </span><span style="color: #b5cea8;">12</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">format</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"Good morning, </span><span style="color: #569cd6;">{</span><span style="color: #9cdcfe;">name</span><span style="color: #569cd6;">}</span><span style="color: #ce9178;">!"</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> } </span><span style="color: #c586c0;">else</span><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">if</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">hour</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;"><</span><span style="color: #d4d4d4;"> </span><span style="color: #b5cea8;">18</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">format</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"Good afternoon, </span><span style="color: #569cd6;">{</span><span style="color: #9cdcfe;">name</span><span style="color: #569cd6;">}</span><span style="color: #ce9178;">!"</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> } </span><span style="color: #c586c0;">else</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">format</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"Good evening, </span><span style="color: #569cd6;">{</span><span style="color: #9cdcfe;">name</span><span style="color: #569cd6;">}</span><span style="color: #ce9178;">!"</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> }</span></div><div><span style="color: #d4d4d4;"> }</span></div><div><span style="color: #d4d4d4;">}</span></div><br /></div><p></p><p>Note how we call the <span style="font-family: courier;">host::hour</span> function and get the hour. Of course here we doing something trivial using standard WebAssembly types like <span style="font-family: courier;">u32</span>, things would become more complicated if the types get more complex.</p><p>The host code needs to provide an implementation of the hour function and inject it via imports:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">hour</span><span style="color: #d4d4d4;">() </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">u32</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">now</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">chrono</span><span style="color: #d4d4d4;">::</span><span style="color: #4ec9b0;">Local</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">now</span><span style="color: #d4d4d4;">();</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">now</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">hour</span><span style="color: #d4d4d4;">()</span></div><div><span style="color: #d4d4d4;">}</span></div><div><span style="color: #d4d4d4;"> </span></div><div><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">main</span><span style="color: #d4d4d4;">() </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Result</span><span style="color: #d4d4d4;"><()> {</span></div></div><span style="color: #d4d4d4;"> </span>...</div></div><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">imports</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">imports</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #ce9178;">"host"</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;">> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #ce9178;">"hour"</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;">> </span><span style="color: #4ec9b0;">Function</span><span style="color: #d4d4d4;">:</span><span style="color: #d4d4d4;">:</span><span style="color: #dcdcaa;">new_typed</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #dcdcaa;">hour</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> },</span></div><div><span style="color: #d4d4d4;"> };</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">instance</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Instance</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">new</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe;">module</span><span style="color: #d4d4d4;">, </span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe;">imports</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;">...</span></div><div><span style="color: #d4d4d4;">} </span></div></div></div><p></p><p>Et voilà!</p><p><span style="font-family: courier;">cargo run -- "JP Moresmau"<br />Language: English<br />Greeting: Good afternoon, JP Moresmau!</span></p><p>All code can be found as before at <a href="https://github.com/JPMoresmau/greet-plugins/tree/wasmer-wai">https://github.com/JPMoresmau/greet-plugins/tree/wasmer-wai</a>.</p><p>Happy WebAssembly and Rust hacking!<br /></p><p><br /></p>JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com0tag:blogger.com,1999:blog-37404288.post-5501192646402671062023-04-10T16:55:00.005+02:002023-04-10T16:55:57.221+02:00Web Assembly Interfaces help integration of WASM libraries<p> In the previous post, I showed how to run plugins generated from Rust code with wasm-bindgen, using the wasmtime crate. I then discovered <a href="https://wasmer.io/">Wasmer</a>, so I rewrote the runtime to use the <a href="https://docs.rs/wasmer/latest/wasmer/">Wasmer API</a>, which is very similar (see <a href="https://github.com/JPMoresmau/greet-plugins/tree/wasmer">https://github.com/JPMoresmau/greet-plugins/tree/wasmer</a>). But I see Wasmer have a lot more tooling available than just a runtime, so let's see how it can help!</p><p>I followed first the tutorial at <a href="https://wasmer.io/posts/wasmer-takes-webassembly-libraries-manistream-with-wai">https://wasmer.io/posts/wasmer-takes-webassembly-libraries-manistream-with-wai</a>. <br /></p><p>You can first define the Web Assembly interface you want to expose in a type of IDL file - think protobuf or Corba, depending on your age :-). Easy enough in our case:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #dcdcaa;">language</span><span style="color: #d4d4d4;">: </span><span style="color: #569cd6;">func</span><span style="color: #d4d4d4;">() -> </span><span style="color: #4ec9b0;">string</span></div><br /><div><span style="color: #dcdcaa;">greet</span><span style="color: #d4d4d4;">: </span><span style="color: #569cd6;">func</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">name: </span><span style="color: #4ec9b0;">string</span><span style="color: #d4d4d4;">) -> </span><span style="color: #4ec9b0;">string</span></div><br /></div><p></p><p>We can put that file (<span style="font-family: courier;">greeter.wai</span>) in our english-rs crate folder, and remove all wasm-bindgen dependencies and related code. We can then use the <span style="font-family: courier;">export!</span> macro of the <a href="https://crates.io/crates/wai-bindgen-rust">wai-bindgen-rust</a> crate to automatically generate a trait that defines both function, and then provide an implementation:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #4ec9b0;">wai_bindgen_rust</span><span style="color: #d4d4d4;">::</span><span style="color: #569cd6;">export</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"greeter.wai"</span><span style="color: #d4d4d4;">);</span></div><br /><div><span style="color: #569cd6;">struct</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Greeter</span><span style="color: #d4d4d4;">;</span></div><br /><div><span style="color: #569cd6;">impl</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">crate</span><span style="color: #d4d4d4;">::</span><span style="color: #4ec9b0;">greeter</span><span style="color: #d4d4d4;">::</span><span style="color: #4ec9b0;">Greeter</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">for</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Greeter</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #6a9955;"> </span><span style="color: #6a9955;">/// The language we greet in.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">language</span><span style="color: #d4d4d4;">() </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">from</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"English"</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> }</span></div><br /><div><span style="color: #6a9955;"> </span><span style="color: #6a9955;">/// Greet the given name.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">greet</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">name</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;">) </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">format</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"Hello, </span><span style="color: #569cd6;">{</span><span style="color: #9cdcfe;">name</span><span style="color: #569cd6;">}</span><span style="color: #ce9178;">!"</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> }</span></div><div><span style="color: #d4d4d4;">}</span></div><br /></div><p></p><p>That's it! The only change from the previous code apart from the impl block is that the name parameter is an owned <span style="font-family: courier;">String</span> and not a <span style="font-family: courier;">&str</span>.</p><p>Then I can publish this library to the wasmer WebAssembly libraries repositories via the <span style="font-family: courier;">cargo-wapm</span> command. It now lives at <a href="https://wapm.io/JPMoresmau/english-rs">https://wapm.io/JPMoresmau/english-rs</a>. You can download the <span style="font-family: courier;">.wasm</span> file from there!<br /></p><p>What about the runtime? There's not a lot of documentation yet because of lot of this is still beta (and the specs of WebAssembly Interfaces and related concepts are still in flux), but it's possible to use the <span style="font-family: courier;">import!</span> macro of the <a href="https://crates.io/crates/wai-bindgen-wasmer">wai-bindgen-wasmer</a> crate to generate code to interact with the module, using the same wai file: we use the same file that defines the interface to both generate the trait we need to implement and the client struct. This is what our greeter now looks like:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #569cd6;">use</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">std</span><span style="color: #d4d4d4;">::</span><span style="color: #d4d4d4;">{</span><span style="color: #4ec9b0;">env</span><span style="color: #d4d4d4;">, </span><span style="color: #4ec9b0;">fs</span><span style="color: #d4d4d4;">};</span></div><br /><div><span style="color: #569cd6;">use</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">anyhow</span><span style="color: #d4d4d4;">::</span><span style="color: #d4d4d4;">{</span><span style="color: #569cd6;">anyhow</span><span style="color: #d4d4d4;">, </span><span style="color: #4ec9b0;">Result</span><span style="color: #d4d4d4;">};</span></div><div><span style="color: #569cd6;">use</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">greeter</span><span style="color: #d4d4d4;">::</span><span style="color: #d4d4d4;">{</span><span style="color: #4ec9b0;">Greeter</span><span style="color: #d4d4d4;">, </span><span style="color: #4ec9b0;">GreeterData</span><span style="color: #d4d4d4;">};</span></div><div><span style="color: #569cd6;">use</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">wasmer</span><span style="color: #d4d4d4;">::</span><span style="color: #d4d4d4;">*;</span></div><div><span style="color: #569cd6;">use</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">wasmer_compiler_llvm</span><span style="color: #d4d4d4;">::</span><span style="color: #4ec9b0;">LLVM</span><span style="color: #d4d4d4;">;</span></div><br /><div><span style="color: #4ec9b0;">wai_bindgen_wasmer</span><span style="color: #d4d4d4;">::</span><span style="color: #569cd6;">import</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"greeter.wai"</span><span style="color: #d4d4d4;">);</span></div><br /><div><span style="color: #6a9955;">/// Greet using all the plugins.</span></div><div><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">main</span><span style="color: #d4d4d4;">() </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Result</span><span style="color: #d4d4d4;"><()> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">args</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">Vec</span><span style="color: #d4d4d4;"><</span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;">> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">env</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">args</span><span style="color: #d4d4d4;">()</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">collect</span><span style="color: #d4d4d4;">();</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">if</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">args</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">len</span><span style="color: #d4d4d4;">() </span><span style="color: #d4d4d4;">!=</span><span style="color: #d4d4d4;"> </span><span style="color: #b5cea8;">2</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">return</span><span style="color: #d4d4d4;"> </span><span style="color: #4fc1ff;">Err</span><span style="color: #d4d4d4;">(</span><span style="color: #569cd6;">anyhow</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"Usage: i18n-greeter <name>"</span><span style="color: #d4d4d4;">));</span></div><div><span style="color: #d4d4d4;"> }</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">compiler_config</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">LLVM</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">default</span><span style="color: #d4d4d4;">();</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">engine</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">EngineBuilder</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">new</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">compiler_config</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">engine</span><span style="color: #d4d4d4;">();</span></div><br /><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">paths</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">fs</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">read_dir</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"./plugins"</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">unwrap</span><span style="color: #d4d4d4;">();</span></div><br /><div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">for</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">path</span><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">in</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">paths</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">path</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">path</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Store</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">new</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe;">engine</span><span style="color: #d4d4d4;">);</span></div><br /><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">module</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Module</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">from_file</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">path</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">path</span><span style="color: #d4d4d4;">())</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><br /><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">imports</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">imports</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;"> {};</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">instance</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Instance</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">new</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe;">module</span><span style="color: #d4d4d4;">, </span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe;">imports</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">env</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">FunctionEnv</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">new</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #4ec9b0;">GreeterData</span><span style="color: #d4d4d4;"> {});</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">greeter</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Greeter</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">new</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe;">instance</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">env</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><br /><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">language</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">greeter</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">language</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">println</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"Language: </span><span style="color: #569cd6;">{</span><span style="color: #9cdcfe;">language</span><span style="color: #569cd6;">}</span><span style="color: #ce9178;">"</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">greeting</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">greeter</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">greet</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe;">args</span><span style="color: #d4d4d4;">[</span><span style="color: #b5cea8;">1</span><span style="color: #d4d4d4;">])</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">println</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"Greeting: </span><span style="color: #569cd6;">{</span><span style="color: #9cdcfe;">greeting</span><span style="color: #569cd6;">}</span><span style="color: #ce9178;">"</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #d4d4d4;"> }</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #4fc1ff;">Ok</span><span style="color: #d4d4d4;">(())</span></div><div><span style="color: #d4d4d4;">}</span></div><br /></div><p></p><p>No more requirement to understand how to call WASM functions and get the return value, the Wasmer WAI generated code does that for you! The API could still be cleaner (maybe without that store parameter everywhere) but the convenience is already a clear win. </p><p>All this code can be found at <a href="https://github.com/JPMoresmau/greet-plugins/tree/wasmer-wai">https://github.com/JPMoresmau/greet-plugins/tree/wasmer-wai</a>.</p><p>Happy WebAssembly hacking!<br /></p>JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com0tag:blogger.com,1999:blog-37404288.post-76729537942945751492023-04-07T15:21:00.004+02:002023-04-07T15:21:48.327+02:00A WebAssembly plugin system<p>Wow, it's been 4 years since I last blogged something! I usually just lurk on social media and I guess I was too busy to find the time to write about something reasonably interesting...</p><p>But hopefully today I did something I can present. I wanted to see how I could use <a href="https://webassembly.org/">WebAssembly</a> to write plugins I could then load dynamically in Rust code. So far, results are good but only work with the <a href="https://rustwasm.github.io/wasm-bindgen/" target="">wasm-bindgen</a> conventions for calling and get results from functions. Still!</p><p>A lot of WebAssembly samples show you how to pass around the basic numeric types like <span style="font-family: courier;">i32</span>, so I wanted something with <span style="font-family: courier;">Strings</span> for a little extra complexity and interest. So our plugins will expose two methods:</p><p>- language takes no argument and return a string indicating which (human, not programming) language the plugin handles</p><p>- greet takes one argument, a person name, and return a greeting in the plugin's language</p><p>As you can see, not too involved, but a nice little use case.</p><h2 style="text-align: left;">The first plugin in Rust</h2><p>I followed the instructions at the <a href="https://rustwasm.github.io/docs/book/">Rust and WebAssembly guide</a> to get started, this was really painless. The Rust code for my first plugin is simply (including some generated code I didn't touch):</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #569cd6;">mod</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">utils</span><span style="color: #d4d4d4;">;</span></div><br /><div><span style="color: #569cd6;">use</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">wasm_bindgen</span><span style="color: #d4d4d4;">::</span><span style="color: #4ec9b0;">prelude</span><span style="color: #d4d4d4;">::</span><span style="color: #d4d4d4;">*;</span></div><br /><div><span style="color: #6a9955;">// When the `wee_alloc` feature is enabled, use `wee_alloc` as the global</span></div><div><span style="color: #6a9955;">// allocator.</span></div><div><span style="color: #d4d4d4;">#[cfg(feature </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #ce9178;">"wee_alloc"</span><span style="color: #d4d4d4;">)]</span></div><div><span style="color: #d4d4d4;">#[</span><span style="color: #dcdcaa;">global_allocator</span><span style="color: #d4d4d4;">]</span></div><div><span style="color: #569cd6;">static</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">ALLOC</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">wee_alloc</span><span style="color: #d4d4d4;">::</span><span style="color: #4ec9b0;">WeeAlloc</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">wee_alloc</span><span style="color: #d4d4d4;">::</span><span style="color: #4ec9b0;">WeeAlloc</span><span style="color: #d4d4d4;">::</span><span style="color: #d4d4d4;">INIT;</span></div><br /><div><span style="color: #6a9955;">/// The language we greet in.</span></div><div><span style="color: #d4d4d4;">#[</span><span style="color: #dcdcaa;">wasm_bindgen</span><span style="color: #d4d4d4;">]</span></div><div><span style="color: #569cd6;">pub</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">language</span><span style="color: #d4d4d4;">() </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">from</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"English"</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;">}</span></div><br /><div><span style="color: #6a9955;">/// Greet the given name.</span></div><div><span style="color: #d4d4d4;">#[</span><span style="color: #dcdcaa;">wasm_bindgen</span><span style="color: #d4d4d4;">]</span></div><div><span style="color: #569cd6;">pub</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">greet</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">name</span><span style="color: #d4d4d4;">: &</span><span style="color: #4ec9b0;">str</span><span style="color: #d4d4d4;">) </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">format</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"Hello, </span><span style="color: #569cd6;">{</span><span style="color: #9cdcfe;">name</span><span style="color: #569cd6;">}</span><span style="color: #ce9178;">!"</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;">}</span></div><br /></div><p></p><p>Running <span style="font-family: courier;">wasm-pack build </span>gives us a <span style="font-family: courier;">.wasm</span> file that exports <span style="font-family: courier;">language</span> and <span style="font-family: courier;">greet</span>, good!</p><h2 style="text-align: left;">The plugin runner in Rust</h2><p>I used the <a href="https://wasmtime.dev/">wasmtime</a> crate to get a runtime engine capable of loading and calling WebAssembly modules. I didn't look for any utility functions and implemented the string handling functions necessary to interact with the <span style="font-family: courier;">wasm-bindgen</span> exposed functions myself.</p><p>So cargo.toml has very few dependencies:</p><p><span style="font-family: courier;">[dependencies]<br />wasmtime = "1.0.0"<br />anyhow = "1.0.70"<br />byteorder = "1.4.3"</span></p><p>The main function is fairly straightforward: it gets the arguments, creates the WASM engine and asks each plugin it can find in a folder to greet the person:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div></div><div><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">main</span><span style="color: #d4d4d4;">() </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Result</span><span style="color: #d4d4d4;"><()> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">args</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">Vec</span><span style="color: #d4d4d4;"><</span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;">> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">env</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">args</span><span style="color: #d4d4d4;">()</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">collect</span><span style="color: #d4d4d4;">();</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">if</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">args</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">len</span><span style="color: #d4d4d4;">() </span><span style="color: #d4d4d4;">!=</span><span style="color: #d4d4d4;"> </span><span style="color: #b5cea8;">2</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">return</span><span style="color: #d4d4d4;"> </span><span style="color: #4fc1ff;">Err</span><span style="color: #d4d4d4;">(</span><span style="color: #569cd6;">anyhow</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"Usage: i18n-greeter <name>"</span><span style="color: #d4d4d4;">));</span></div><div><span style="color: #d4d4d4;"> }</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">engine</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Engine</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">default</span><span style="color: #d4d4d4;">();</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">linker</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Linker</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">new</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe;">engine</span><span style="color: #d4d4d4;">);</span></div><br /><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">paths</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">fs</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">read_dir</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"./plugins"</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">unwrap</span><span style="color: #d4d4d4;">();</span></div><br /><div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">for</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">path</span><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">in</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">paths</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">path</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">path</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">module</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Module</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">from_file</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe;">engine</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">path</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">path</span><span style="color: #d4d4d4;">())</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">runtime</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Runtime</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">new</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe;">engine</span><span style="color: #d4d4d4;">, </span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe;">linker</span><span style="color: #d4d4d4;">, </span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe;">module</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">language</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">runtime</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa; text-decoration: underline;">language</span><span style="color: #d4d4d4;">()</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">println</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"Language: </span><span style="color: #569cd6;">{</span><span style="color: #9cdcfe;">language</span><span style="color: #569cd6;">}</span><span style="color: #ce9178;">"</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">greeting</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">runtime</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa; text-decoration: underline;">greet</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe;">args</span><span style="color: #d4d4d4;">[</span><span style="color: #b5cea8;">1</span><span style="color: #d4d4d4;">])</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">println</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"Greeting: </span><span style="color: #569cd6;">{</span><span style="color: #9cdcfe;">greeting</span><span style="color: #569cd6;">}</span><span style="color: #ce9178;">"</span><span style="color: #d4d4d4;">);</span></div><div><span style="color: #d4d4d4;"> }</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #4fc1ff;">Ok</span><span style="color: #d4d4d4;">(())</span></div><div><span style="color: #d4d4d4;">}</span></div></div><p></p><p>The magic is inside the Runtime struct, that actually handles the nitty-gritty of initializing all the necessary things for wasmtime to do its thing:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #569cd6;">struct</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Runtime</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">store</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">Store</span><span style="color: #d4d4d4;"><()>,</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">memory</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">Memory</span><span style="color: #d4d4d4;">,</span></div><div><span style="color: #6a9955;"> </span><span style="color: #6a9955;">/// Pointer to currently unused memory.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">pointer</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">usize</span><span style="color: #d4d4d4;">,</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">language</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">TypedFunc</span><span style="color: #d4d4d4;"><</span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">, ()>,</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">greet</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">TypedFunc</span><span style="color: #d4d4d4;"><(</span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">, </span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">, </span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">), ()>,</span></div><div><span style="color: #d4d4d4;">}</span></div></div><p></p><p>We have to manage the WebAssembly linear memory ourselves so we do it very simplify by keeping a pointer to where in the memory we can put stuff.</p><p>Initializing the runtime:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">new</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">engine</span><span style="color: #d4d4d4;">: &</span><span style="color: #4ec9b0;">Engine</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">linker</span><span style="color: #d4d4d4;">: &</span><span style="color: #4ec9b0;">Linker</span><span style="color: #d4d4d4;"><()>, </span><span style="color: #9cdcfe;">module</span><span style="color: #d4d4d4;">: &</span><span style="color: #4ec9b0;">Module</span><span style="color: #d4d4d4;">) </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Result</span><span style="color: #d4d4d4;"><</span><span style="color: #569cd6;">Self</span><span style="color: #d4d4d4;">> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Store</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">new</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">engine</span><span style="color: #d4d4d4;">, ());</span></div><br /><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">instance</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">linker</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">instantiate</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">module</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><br /><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">memory</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">instance</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">get_memory</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #ce9178;">"memory"</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">ok_or</span><span style="color: #d4d4d4;">(</span><span style="color: #4ec9b0;">anyhow</span><span style="color: #d4d4d4;">::</span><span style="color: #569cd6;">format_err</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"failed to find `memory` export"</span><span style="color: #d4d4d4;">))</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">language</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">instance</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">get_func</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #ce9178;">"language"</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">ok_or</span><span style="color: #d4d4d4;">(</span><span style="color: #4ec9b0;">anyhow</span><span style="color: #d4d4d4;">::</span><span style="color: #569cd6;">format_err</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #ce9178;">"`language` was not an exported function"</span></div><div><span style="color: #d4d4d4;"> ))</span><span style="color: #d4d4d4;">?</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">typed</span><span style="color: #d4d4d4;">::</span><span style="color: #d4d4d4;"><</span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">, (), </span><span style="color: #9cdcfe;">_</span><span style="color: #d4d4d4;">>(</span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">greet</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">instance</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">get_func</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #ce9178;">"greet"</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">ok_or</span><span style="color: #d4d4d4;">(</span><span style="color: #4ec9b0;">anyhow</span><span style="color: #d4d4d4;">::</span><span style="color: #569cd6;">format_err</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">(</span><span style="color: #ce9178;">"`greet` was not an exported function"</span><span style="color: #d4d4d4;">))</span><span style="color: #d4d4d4;">?</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">typed</span><span style="color: #d4d4d4;">::</span><span style="color: #d4d4d4;"><(</span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">, </span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">, </span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">), (), </span><span style="color: #9cdcfe;">_</span><span style="color: #d4d4d4;">>(</span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe; text-decoration: underline;">store</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><br /><div><span style="color: #d4d4d4;"> </span><span style="color: #4fc1ff;">Ok</span><span style="color: #d4d4d4;">(</span><span style="color: #569cd6;">Self</span><span style="color: #d4d4d4;"> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">store</span><span style="color: #d4d4d4;">,</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">memory</span><span style="color: #d4d4d4;">,</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">pointer</span><span style="color: #d4d4d4;">: </span><span style="color: #b5cea8;">0</span><span style="color: #d4d4d4;">,</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">language</span><span style="color: #d4d4d4;">,</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">greet</span><span style="color: #d4d4d4;">,</span></div><div><span style="color: #d4d4d4;"> })</span></div><div><span style="color: #d4d4d4;"> }</span></div></div><p></p><p>With this we can do our own very basic memory management, which means reserving an area of memory, for example to read and write strings as UTF8 byte arrays, using the wasm-bindgen conventions:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #6a9955;"> </span><span style="color: #6a9955;">/// Get a new pointer to store the given size in memory.</span></div><div><span style="color: #6a9955;"> </span><span style="color: #6a9955;">/// Grows memory if needed.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa; text-decoration: underline;">new_pointer</span><span style="color: #d4d4d4;">(&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">size</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">usize</span><span style="color: #d4d4d4;">) </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Result</span><span style="color: #d4d4d4;"><</span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">current</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">pointer</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">pointer</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4; text-decoration: underline;">+=</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">size</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #c586c0;">while</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">pointer</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">></span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">memory</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">data_size</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">store</span><span style="color: #d4d4d4;">) {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">memory</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">grow</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #b5cea8;">1</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> }</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #4fc1ff;">Ok</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">current</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">as</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> }</span></div><br /><div><span style="color: #6a9955;"> </span><span style="color: #6a9955;">/// Reset pointer, so memory can get overwritten.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa; text-decoration: underline;">reset_pointer</span><span style="color: #d4d4d4;">(&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">) {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">pointer</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #b5cea8;">0</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> }</span></div><br /><div><span style="color: #6a9955;"> </span><span style="color: #6a9955;">/// Read string from memory.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">read_string</span><span style="color: #d4d4d4;">(&</span><span style="color: #569cd6;">self</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">length</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">) </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Result</span><span style="color: #d4d4d4;"><</span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;">> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">contents</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">vec</span><span style="color: #569cd6;">!</span><span style="color: #d4d4d4;">[</span><span style="color: #b5cea8;">0</span><span style="color: #d4d4d4;">; </span><span style="color: #9cdcfe;">length</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">as</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">usize</span><span style="color: #d4d4d4;">];</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">memory</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">read</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">as</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">usize</span><span style="color: #d4d4d4;">, </span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">contents</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #4fc1ff;">Ok</span><span style="color: #d4d4d4;">(</span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;">::</span><span style="color: #dcdcaa;">from_utf8</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe; text-decoration: underline;">contents</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> }</span></div><br /><div><span style="color: #6a9955;"> </span><span style="color: #6a9955;">/// Read bounds from memory.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa;">read_bounds</span><span style="color: #d4d4d4;">(&</span><span style="color: #569cd6;">self</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;">: </span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">) </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Result</span><span style="color: #d4d4d4;"><(</span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">, </span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">)> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">buffer</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> [</span><span style="color: #b5cea8;">0</span><span style="color: #b5cea8;">u8</span><span style="color: #d4d4d4;">; </span><span style="color: #b5cea8;">8</span><span style="color: #d4d4d4;">];</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">memory</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">read</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">as</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">usize</span><span style="color: #d4d4d4;">, </span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe; text-decoration: underline;">buffer</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">start</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> (</span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe; text-decoration: underline;">buffer</span><span style="color: #d4d4d4;">[</span><span style="color: #b5cea8;">0</span><span style="color: #d4d4d4;">..</span><span style="color: #b5cea8;">4</span><span style="color: #d4d4d4;">])</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa; text-decoration: underline;">read_i32</span><span style="color: #d4d4d4;">::</span><span style="color: #d4d4d4;"><</span><span style="color: #4ec9b0;">LittleEndian</span><span style="color: #d4d4d4;">>()</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">length</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> (</span><span style="color: #d4d4d4;">&</span><span style="color: #9cdcfe; text-decoration: underline;">buffer</span><span style="color: #d4d4d4;">[</span><span style="color: #b5cea8;">4</span><span style="color: #d4d4d4;">..</span><span style="color: #d4d4d4;">])</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa; text-decoration: underline;">read_i32</span><span style="color: #d4d4d4;">::</span><span style="color: #d4d4d4;"><</span><span style="color: #4ec9b0;">LittleEndian</span><span style="color: #d4d4d4;">>()</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #4fc1ff;">Ok</span><span style="color: #d4d4d4;">((</span><span style="color: #9cdcfe;">start</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">length</span><span style="color: #d4d4d4;">))</span></div><div><span style="color: #d4d4d4;"> }</span></div><br /><div><span style="color: #6a9955;"> </span><span style="color: #6a9955;">/// Write string into memory.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa; text-decoration: underline;">write_string</span><span style="color: #d4d4d4;">(&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">str</span><span style="color: #d4d4d4;">: &</span><span style="color: #4ec9b0;">str</span><span style="color: #d4d4d4;">) </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Result</span><span style="color: #d4d4d4;"><(</span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">, </span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">)> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">data</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">str</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">as_bytes</span><span style="color: #d4d4d4;">();</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa; text-decoration: underline;">new_pointer</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">data</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">len</span><span style="color: #d4d4d4;">())</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">memory</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">write</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">as</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">usize</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">data</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #4fc1ff;">Ok</span><span style="color: #d4d4d4;">((</span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">str</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">len</span><span style="color: #d4d4d4;">() </span><span style="color: #569cd6;">as</span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">i32</span><span style="color: #d4d4d4;">))</span></div><div><span style="color: #d4d4d4;"> }</span></div><br /><br /></div><p></p><p>Basically we pass two <span style="font-family: courier;">i32</span> when we need to transfer a string, the offset and length that we use on the linear memory to read or write the bytes.</p><p>Using these, wrapping our plugin functions is easy:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Menlo, Monaco, 'Courier New', monospace; font-size: 12px; font-weight: normal; line-height: 18px; white-space: pre;"><div><span style="color: #6a9955;">/// Call language function.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa; text-decoration: underline;">language</span><span style="color: #d4d4d4;">(&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">) </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Result</span><span style="color: #d4d4d4;"><</span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;">> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa; text-decoration: underline;">new_pointer</span><span style="color: #d4d4d4;">(</span><span style="color: #b5cea8;">16</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">language</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">call</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">store</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> (</span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">length</span><span style="color: #d4d4d4;">) </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">read_bounds</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">s</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">read_string</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">length</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa; text-decoration: underline;">reset_pointer</span><span style="color: #d4d4d4;">();</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #4fc1ff;">Ok</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">s</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> }</span></div><br /><div><span style="color: #6a9955;"> </span><span style="color: #6a9955;">/// Call greet function.</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">fn</span><span style="color: #d4d4d4;"> </span><span style="color: #dcdcaa; text-decoration: underline;">greet</span><span style="color: #d4d4d4;">(&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">name</span><span style="color: #d4d4d4;">: &</span><span style="color: #4ec9b0;">str</span><span style="color: #d4d4d4;">) </span><span style="color: #d4d4d4;">-></span><span style="color: #d4d4d4;"> </span><span style="color: #4ec9b0;">Result</span><span style="color: #d4d4d4;"><</span><span style="color: #4ec9b0;">String</span><span style="color: #d4d4d4;">> {</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa; text-decoration: underline;">new_pointer</span><span style="color: #d4d4d4;">(</span><span style="color: #b5cea8;">16</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> (</span><span style="color: #9cdcfe;">start</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">length</span><span style="color: #d4d4d4;">) </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa; text-decoration: underline;">write_string</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">name</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">greet</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">call</span><span style="color: #d4d4d4;">(</span><span style="color: #d4d4d4;">&</span><span style="color: #569cd6;">mut</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">store</span><span style="color: #d4d4d4;">, (</span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">start</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">length</span><span style="color: #d4d4d4;">))</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> (</span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">length</span><span style="color: #d4d4d4;">) </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">read_bounds</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6;">let</span><span style="color: #d4d4d4;"> </span><span style="color: #9cdcfe;">s</span><span style="color: #d4d4d4;"> </span><span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">read_string</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">offset</span><span style="color: #d4d4d4;">, </span><span style="color: #9cdcfe;">length</span><span style="color: #d4d4d4;">)</span><span style="color: #d4d4d4;">?</span><span style="color: #d4d4d4;">;</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #569cd6; text-decoration: underline;">self</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa; text-decoration: underline;">reset_pointer</span><span style="color: #d4d4d4;">();</span></div><div><span style="color: #d4d4d4;"> </span><span style="color: #4fc1ff;">Ok</span><span style="color: #d4d4d4;">(</span><span style="color: #9cdcfe;">s</span><span style="color: #d4d4d4;">)</span></div><div><span style="color: #d4d4d4;"> }</span></div></div><p></p><p>So if we copy the wasm file compiled from our first plugin into the <span style="font-family: courier;">plugins</span> directory and run the program with my name:</p><p><span style="font-family: courier;">cargo run "JP Moresmau"<br /> Finished dev [unoptimized + debuginfo] target(s) in 0.08s<br /> Running `target/debug/i18n-greeter 'JP Moresmau'`<br />Language: English<br />Greeting: Hello, JP Moresmau!</span></p><h2 style="text-align: left;">Further steps</h2><p style="text-align: left;">Of course now what would be very cool would be to be able to write plugins in other languages, but for example it looks like Go, even with <a href="https://tinygo.org/docs/guides/webassembly/">TinyGo</a>, is still very much tied to a Javascript runtime. Maybe the wasm-bindgen conventions will be ported to other languages than Rust in the future?</p><h2 style="text-align: left;">Trying it yourself<br /></h2><p>All the code can be found at <a href="https://github.com/JPMoresmau/greet-plugins">https://github.com/JPMoresmau/greet-plugins</a>.<br /></p>JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com0tag:blogger.com,1999:blog-37404288.post-80847554318901903782019-06-02T15:46:00.000+02:002019-06-02T15:46:29.895+02:00Rust interviews questionsA few days ago <a href="https://glenmccallum.com/2019/05/14/senior-developers-rejected-jobs/">this blog entry</a> sparkled a big discussion on <a href="https://www.reddit.com/r/programming/comments/bombvw/senior_developers_are_getting_rejected_for_jobs/">Reddit</a>, and it got me thinking that I just couldn't be relying on my 20 years experience in professional programming if I ever needed another job. I need to practice small, low level coding problems.<br />
<br />
So of course I decided to use Rust. I'm putting my code on <a href="https://github.com/JPMoresmau/rust-interview">Github </a>so feel free to have a look if you're interested. I tried to not look at existing resources, I'm sure you can find plenty of Rust interview answers and toy problems if you look around :-).<br />
<br />
First I tackled some array operations like finding and removing duplicates, implementing <a href="https://en.wikipedia.org/wiki/Quicksort">quicksort</a>, etc. I just used <span style="font-family: Courier New, Courier, monospace;">Vec<i32></i32></span> and implemented the algorithms only. In some cases of course a Set is what makes sense (finding duplicates) so I just used another vector as a set, of course implementing an efficient integer set would a good problem too. Here I didn't run into any major issue.<br />
<br />
The second (and last, for now) thing I did is to implement a Linked List from scratch. The list owns its values of course, so implementing pop/push methods and iterator support proved "interesting"! Now I really had to understand the borrow checker, owning vs referencing, etc. I even tried to do thing using unsafe code like <a href="https://doc.rust-lang.org/std/boxed/struct.Box.html#method.into_raw">Box::into_raw</a>! And? Well, my program... crashed. I think I'll tread carefully if ever I need to use unsafe.<br />
In the end, <a href="https://doc.rust-lang.org/std/mem/fn.replace.html">mem::replace</a> was my savior. I could get and own a cell from my list by swapping it with an empty cell, do whatever I needed to do, and reset the cell to a proper value. That worked a charm!<br />
<br />
So, I'm still happy enough with Rust. The error messages are for the most part clear enough to be able to understand, I do find enough on the web between the <a href="https://www.rust-lang.org/learn">official docs</a> and <a href="https://stackoverflow.com/search?q=rust">Stack Overflow</a> to save me from being stuck. Rust is good!<br />
<br />
Happy Rust Hacking!<br />
<br />JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com0tag:blogger.com,1999:blog-37404288.post-44290297342637754142019-03-10T14:15:00.000+01:002019-03-10T14:15:58.119+01:00Rust for linear algebra and neural networks<span style="font-family: inherit;">From time to time I do a little bit of fooling around with algorithms and AI, I suppose it's a break from what I do in work, which is more about building business applications. I don't have a real goal in mind, I don't think I'm going to stumble about an algorithm for <a href="https://en.wikipedia.org/wiki/Artificial_general_intelligence">general intelligence</a> or something, but I like to understand how things work at a low level.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">So I've writing a bit of Rust code to implement some linear algebra (some matrix and vector operations) and use that to build some small neural networks. You can find the current state of that <a href="https://github.com/JPMoresmau/rust-linear">code </a>here. Of course, if you really want to use production ready code for linear algebra you should check <a href="https://www.rustsim.org/">https://www.rustsim.org/</a> and if neural networks and AI in Rust are what you're after please check <a href="https://github.com/AtheMathmo/rusty-machine">https://github.com/AtheMathmo/rusty-machine</a>. Note that rusty-machine is still in a state of flux since it's using its own linear algebra library but there's <a href="https://github.com/AtheMathmo/rusty-machine/issues/199">talk </a>about using <a href="https://nalgebra.org/">nalgebra </a>from <a href="https://www.rustsim.org/">rustsim </a>hopefully that will bring together all these initiatives in one unified Rust platform for numeric operations and AI.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">I have to say I'm still impressed with Rust. I like that you can do a lot of things with a functional approach, while still being able to update variable values or do other imperative operations when it makes sense. It does feel a lot like "pragmatic Haskell" at times. So far I haven't struggled with the borrow checker, but the code I've written so far is fairly simple and self-contained.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">As always, I'm still wondering if there are some projects that it would make sense for me to contribute too. I'm certainly not a AI or math expert, but if you have a Rust project that could do with some contributions, let me know!</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Happy Rust Hacking!</span>JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com0tag:blogger.com,1999:blog-37404288.post-29479091290188497082019-02-10T16:48:00.000+01:002019-02-10T16:49:42.139+01:00Rust helped me escape Cthulhu!I just downloaded the game <a href="https://play.google.com/store/apps/details?id=com.puzzle.comix&hl=en_US">Lovecraft Quest</a> on my Android tablet, and was enjoying the puzzles, until I got to the final one, and that got me, ahem, *frustrated*. I found a <a href="http://www.marvingames.com/16778-lovecraft-quest-a-comix-game-walkthrough-all-3-parts/">walk-through</a> where the reviewers admitted to more or less clicking randomly on the screen for one hour until it worked:<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/nlfrLEYNxm0/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/nlfrLEYNxm0?feature=player_embedded&t=795" width="320"></iframe></div>
<br />
I actually managed to solve the puzzle the first time quickly with sheer luck, but on the second run I was stumped. So I followed the only reasonable course of action: let the computer solve the puzzle for me!<br />
<br />
The puzzle is as follows: there is a square (4 rows, 4 columns) of gold bars that can be either in horizontal or vertical position. You can make one bar pivot, but when you do that all the bars in the same row and column pivot too! You need to get all bars to the vertical position for the door to open to reveal... Cthulhu itself!<br />
<br />
So let us use Rust for this puzzle, and see if a brute force approach can work.
<br />
Let's first define a few types. A grid is a 4x4 array of booleans and a position is a tuple. We'll have positions ranging from (0,0) to (3,3). A grid is solved is all elements are true (so true means vertical position for a bar).<br />
<script src="https://gist.github.com/JPMoresmau/663e0c8148fda818563cd727153dda06.js"></script>
Then let's implement the swapping operation. We'll take one Grid reference and return a new Grid with the proper bars swapped.<br />
<script src="https://gist.github.com/JPMoresmau/943a4e4b1546ed44896b986b81800b62.js"></script>
Note that we swap again the position we chose since we swapped it twice already, one for the columns and once for the rows. Not very elegant maybe but simple<br />
<br />
The main solving function is as follows: we're given a grid and we return a list of positions to swap to solve the puzzle. If the grid is already solved we return an empty vec. Otherwise we keep two structures: a map of grids to a vector of positions (showing how we got to that particular grid from the start grid) and a list of grids to process. We use a Deque here so we can put new grids at the end but process grids from the start, to do a breadth-first traversal of all possible grids.<br />
<script src="https://gist.github.com/JPMoresmau/4c301e6e03bd1b4da283b5cca17ac63e.js"></script>
The try_all function will do all possible swaps on the given grid and store all the results with the path used to get to them. We store for all grids the shortest path we used to get to it. New grids get added at the end of the todo list. We return the path if we end up with a winning grid.<br />
<script src="https://gist.github.com/JPMoresmau/8763a50adb796214f0df3d76b64b684d.js"></script>
To simplify calling the program, we'll just pass a string made of 16 characters, with 1s and 0s indicating the position of the bars we see on the screen. Parsing this string into a Grid instance is easy enough with the help of chunks():<br />
<script src="https://gist.github.com/JPMoresmau/2f8c58c8973de1df8cea65f419720539.js"></script>
And we can pass then that string as a argument to our binary program, and print the result:<br />
<script src="https://gist.github.com/JPMoresmau/e6d4ee734513ea7156de94d7eee2670c.js"></script>
In my case, the program ran and gave me the solution: only 12 moves!<br />
<br />
This Rust code is probably fairly naive in places, since I only have a couple of weeks of evenings learning it. I'm not too happy with the number of clone() calls I had to add to make the borrow checker happy, on top of the ones I thought I needed anyway. Maybe immutable structures would in fact be easier to use here! Of course this code could be made parallel but since it gave me my answer in a few seconds I didn't need to improve it further.<br />
<br />
Happy Rust Hacking!<br />
<br />
<br />JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com1tag:blogger.com,1999:blog-37404288.post-28446466194197961822019-02-03T14:18:00.000+01:002019-02-09T11:44:49.718+01:00Pedal to the Metal: starting with Rust<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
After hearing for
years about Rust, I finally decided to give it a go. This is my journey so far</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Did the whole Rust book at </span><a href="https://doc.rust-lang.org/book/"><span style="font-family: "calibri"; font-size: 11.0pt;">https://doc.rust-lang.org/book/</span></a></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Did the simple Rustlings
exercises at </span><a href="https://github.com/rustlings/rustlings"><span style="font-family: "calibri"; font-size: 11.0pt;">https://github.com/rustlings/rustlings</span></a></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Started the Exercism Rust
track at </span><a href="https://exercism.io/my/tracks/rust"><span style="font-family: "calibri"; font-size: 11.0pt;">https://exercism.io/my/tracks/rust</span></a></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I'm currently using
Visual Studio Code with the <a href="https://marketplace.visualstudio.com/items?itemName=rust-lang.rust">Rustextension</a>.
It's sufficient for the little exercises I've been doing so far, but I notices
that code completion is not very accurate and does not seem to give me
everything that's possible.<span style="mso-spacerun: yes;"> </span>I've
installed the IntelliJ <a href="https://intellij-rust.github.io/">Rust extension</a> in my community IDEA, I'll see how well it works when I start doing bigger
projects.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Rust is obviously a
different kettle of fish than the last language I picked up, which was <a href="http://jpmoresmau.blogspot.com/2018/12/1-2-3-go.html">Go</a>. It's
a lot more complex and rich and hopefully powerful, and consequently takes a lot more time to master.
It feels a bit like a cross between C for the low level aspects and Haskell for
the structural and functional parts. I was a bit surprised straight away to
read about <a href="https://doc.rust-lang.org/rust-by-example/variable_bindings/scope.html">variable shadowing</a> since it seems to take away the security somehow (you cannot mutate a variable
but you can redefine it), but from what I read it doesn't seem to be an issue
practically since the compiler holds your hand at all time.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Most languages I've
worked with have a Garbage Collector, so I'd say dealing with the borrow
checker is going to be fun in larger code bases, but so far in the small
exercises I've done it hasn't been an issue.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Hopefully I can get
productive with Rust quickly, I'd like to contribute to projects like <a href="https://crates.io/crates/rusty-machine">Rusty-Machine</a> to do neural networks and ML with Rust, which would seem a perfect match with
its performance and safety. We'll see!</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Happy Rust Hacking!</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com1tag:blogger.com,1999:blog-37404288.post-67773846483275624692019-01-12T11:41:00.000+01:002019-01-12T11:41:10.284+01:00A little Go project: Cassandra + Elastic + GraphQL<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I've put a<a href="https://github.com/JPMoresmau/nsrep"> littleproject</a> on Github written in <a href="https://golang.org/">Go</a>. It's a little database system that allows
writing and reading objects, and do <a href="https://graphql.org/learn/">GraphQL </a>queries on them. The main principle
is that objects have a arbitrary String ID that is like a path, separated by
slashes, and deleting an item with a given ID deletes also items with
"children" ids.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
For fun, I used
<a href="http://cassandra.apache.org/">Cassandra </a>as the main data store, since it's optimized for fast writes and
loads of data; we never delete anything, we just keep every state of an object
as history. Text searches are done on the last version of objects using
<a href="https://www.elastic.co/products/elasticsearch">Elastic</a>. GraphQL support allows to follow object namespace relationships.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I've used the
following main libraries:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><a href="https://github.com/gocql/gocql"><span style="font-family: Calibri; font-size: 11.0pt;">https://github.com/gocql/gocql</span></a><span style="font-family: Calibri; font-size: 11.0pt;"> for the Cassandra client</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><a href="https://github.com/olivere/elastic"><span style="font-family: Calibri; font-size: 11.0pt;">https://github.com/olivere/elastic</span></a><span style="font-family: Calibri; font-size: 11.0pt;"> for the Elastic client</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><a href="https://github.com/graphql-go/graphql"><span style="font-family: Calibri; font-size: 11.0pt;">https://github.com/graphql-go/graphql</span></a><span style="font-family: Calibri; font-size: 11.0pt;"> for GraphQL.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I also provide
Dockerfiles to deploy either just the database systems for testing, or the
whole application.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I'm not sure this
can be useful as is for anybody, but this was for me to try Go on a real-life
project. Apart from the gotchas I've <a href="http://jpmoresmau.blogspot.com/2019/01/go-gotchas.html">wrote about before</a>, it was a rather
painless and enjoyable experience.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Happy Go Hacking!</div>
<br />JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com0tag:blogger.com,1999:blog-37404288.post-50918989448729464882019-01-05T12:33:00.001+01:002019-01-29T22:05:40.948+01:00Go GotchasAs I wrote my first non trivial project in Go, I had a few head-scratching moments. Nothing that a quick search could clear up, but worth keeping in mind...<br />
<br />
<h4>
<a href="https://golang.org/doc/faq#nil_error">Nil is not nil.</a> </h4>
You may run into that one quickly enough in tests. An instance of an interface can have a nil value but since it carries the interface type information, it does not equal to nil!! A lot has been written about it so make sure to read about it!<br />
<br />
<h4>
Redeclarations</h4>
The fact that you can use := to declare variables easily is a double edged sword, as declaring a variable with the same name as a previously declared one is allowed if it's in a different scope, and it shadows the original variable. So:<br />
<br />
<br />
<script src="https://gist.github.com/JPMoresmau/ed8da9eeda3286629eaf01d1e7b520a7.js"></script>
<br />
This code will only print True at the end. The "err" variable inside the if block has been redefined so even though it's not nil inside the if block, it IS nil once you exit it! So if you want to chain several operations that may return errors and then at the end handle whatever error occurred in a consistent way, you have to be careful.<br />
<br />
<h4>
Closures</h4>
This one gave more some Javascript PTSD:<br />
<br />
<script src="https://gist.github.com/JPMoresmau/9ddb752da350ebeab3e49fc8cdc57ce5.js"></script>
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: inherit;">This prints :</span><br />
<span style="font-family: inherit;">4</span><br />
<span style="font-family: inherit;">4</span><br />
<span style="font-family: inherit;">4</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Which was not what I expected! One easy fix is to add a line saying i:=i before creating the func in the first loop, which by virtue of the previous gotcha redeclares a variable in the scope of the for loop body.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h4>
<span style="font-family: inherit;">Errors</span></h4>
<span style="font-family: inherit;">I'm not convinced in the end by the standard error reporting in Go. A lot has been written on that topic too. I feel that the code is peppered with error handling that gets in the way of readability and implementing cleanly the business logic. I'm sure there are some design tricks you can use, but you shouldn't have to resort to tricks to do clean error handling in a language, because at the end of the day it's how you handle errors that make up the quality of your software, not only how you handle the happy path.</span><br />
<span style="font-family: inherit;"><br /></span>
These things you can watch out for and probably code around pretty easily, but they do impact a bit the ease of use of Go, even though I still think it's quite a simple elegant language to quickly be productive in.<br />
<br />
Happy Go Hacking!JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com0tag:blogger.com,1999:blog-37404288.post-19441113863174627142018-12-15T11:20:00.001+01:002018-12-15T11:20:23.065+01:001, 2, 3, Go!<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">OK, the title is an easy joke, but that's what happens when you give a silly name to a language… So I've started looking at <a href="https://golang.org/">Golang</a>, since it seems to be the newish (ok, ok, not that new) kid on the block. And in fact the name is not an issue: I usually type "golang" to prefix my searches on the web and it works pretty well.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="white-space: pre;"> </span>- I <a href="https://golang.org/doc/install">installed </a>Go and the first striking thing are the instructions to read "<a href="https://golang.org/doc/code.html">how to write Go code</a>". Having one way to structure the code (one workspace containing source control repositories, each containing packages) makes it very easy to get started - compared say to Java where different IDEs may have different approaches to what a "project" is. I wonder how this structure works when you write Go professionally, and you want to distinguish open sources dependencies from proprietary code. Also coming from Java where you're used to structure classes into packages, seeing Go repositories with a huge number of files inside one folder makes me shudder, but we'll see over time.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="white-space: pre;"> </span>- I started of course with the <a href="https://tour.golang.org/list">Tour of Go</a>. In a couple of evenings I went through it, proving how easy and small the base language is. Encouraging!</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="white-space: pre;"> </span>- I then started to get my hands dirty by writing a little app that hooks together JSON web services, Cassandra, Elastic Search and Docker. It was fairly easy to find libraries and manage to put them to use. I'll probably push that code to Github at some stage after some cleaning up.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">So Go delivers on the promise to be a language easy to pick up and easy to get started. You can get from zero to productive in a few hours.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">These are the features that stood up for me:</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="white-space: pre;"> </span>- <b>Lack of generics</b>. It's a trope in the computing world "Go doesn't have generics, ha ha ha". Of course straight away you see that arrays/slices and maps are typed, so already the basic use case for generics (safety in collections) is taken care of. Then you have functions as first-class citizens and interfaces, so there are plenty of techniques you can use to go around the lack of generics, so I'm not sure it's such a huge problem.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="white-space: pre;"> </span>- <b>Interfaces</b>. It's interesting that there is no explicit declaration that a type implements an interface. If you define a Duck interface with a Quack method, every type that you use as the receiver for an implementation of a Quack function is considered a Duck. This is nice, but still tooling support to find out all types that implement a given interface will be a must ("OK, this function expects anything implementing Duck, what implementation can I use?").</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="white-space: pre;"> </span>- <b>Pointers</b>. I'm at the core a Java developer, so the idea of pointers makes me a bit uneasy. But I breathed easier when I saw "no pointer arithmetic". In fact, pointers in Go just make explicit if you pass a struct by value or by reference. I feel this adds a level of clarity that is worth the extra syntax and the initial repulsion the word "pointer" may awake in some.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><span style="white-space: pre;"> </span>- <b>Errors</b>. Go recommends a style where a function that may fail should return a tuple: normal value, error. The calling code can decide to ignore the error or deal with it. This is reminiscent of <a href="http://hackage.haskell.org/package/base-4.12.0.0/docs/Data-Either.html">Either </a>in Haskell. I'm still not sold that this is a better system than checked exceptions (but I seem to be the only one liking checked exceptions it seems), but I'll need more practice before I make my mind up.</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">So all in all I like a lot of design decisions that were made in Go, and it certainly makes for a pleasant programming experience. I ran into a few gotchas that I'll cover maybe in later posts, but nothing too stressful. So far, I have to say I enjoyed the experience!</span><br />
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;">Happy Go Hacking!</span><br />
<div>
<br /></div>
JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com2tag:blogger.com,1999:blog-37404288.post-4699352853411850272018-12-08T16:20:00.000+01:002018-12-08T16:20:09.105+01:00A little RPG in React Native<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
I
built a <a href="https://github.com/JPMoresmau/anthea-quest-rpg">little role-playing game</a> in <a href="https://facebook.github.io/react-native/">React Native</a>. I tried to build the type of
game I like playing, with quests and interactions with non-playing characters,
while learning some mobile UI programming tricks.<span style="mso-spacerun: yes;"> </span>These are some of the main points that I took
away:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">The initial setup was a bit
involved, probably because I'm not used to develop JavaScript
applications. Getting the right code to properly do the plumbing between
React, Redux,<span style="mso-spacerun: yes;"> </span>R<a href="https://reactnavigation.org/">eact Navigation</a>,
tests, etc. was a bit of a hassle. Obviously this has to be done once and
then you can concentrate on writing the functional code. As I noted
recently, then upgrading to newer versions of your dependencies can be a
hassle!</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">I was happy with my
declarative approach to the game world. I don't think it would be
manageable to do otherwise.<span style="mso-spacerun: yes;"> </span>I
define as JavaScript structures the world itself: the different places and
how they communicate, the items they contain, etc., the non-playing
characters and the possible interactions you can have with them, items and
their characteristics, monsters and spells. This gives me an immutable
representation of the start world, and the state of an actual game just
has to store the modifications from that initial state</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">This of course could mean
that I (or somebody else) could reuse the game mechanics for a totally
different adventure!</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">I of course used of <a href="https://redux.js.org/">Redux </a>to
manage state. The state is only modified by Redux reducers, so everything
the player does in the UI just becomes actions, not explicit state
management operations. I combine several reducers, each reducer being
responsible for modifying one section of the global state, which makes
each reducer simpler, but fragment the code. For example when the player
takes an item from a room, one reducer adds the item to the inventory,
another removes the item from the room, so maybe there's some potential
for bugs there</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">I found the integration
between React and Redux easy to use: you simply define the mapping between
the state and components properties, and between component interactions
and Redux actions.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">I abused a little bit Redux
to implement combat: combat holds a kind a mini state for each fighting
round, and then the result of a fighting round both is represented in the
UI ("the rats bite you for 2 damage"), and impacts the global
world.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">I didn't spend too much time
on the UI, just using basic React Native components. I found some of the
layout hard to do correctly, and styling components is not intuitive when
you're not a CSS expert. Maybe I should take a CSS/Design course, because
you can tell from my UI designs that I'm a programmer :-)</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">All in all, writing in modern
JavaScript was nicer than expected, even though I think using a stronger
typed language would be better. If I had to do another similar project I
would probably try to use Typescript. I liked especially the <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax#Spread_in_object_literals">object spreads</a> to update parts of structures and the use of <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/const">consts</a>.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">I used Expo to test on a
tablet, I didn't try to actually generate a final application outside of
Expo, I haven't looked at all on how to distribute the game as an app.</span></li>
</ul>
<div>
<span style="font-family: Calibri;"><span style="font-size: 14.6667px;"><br /></span></span></div>
<div>
<span style="font-family: Calibri;"><span style="font-size: 14.6667px;">All in all, if was an enjoyable experience, but of course designing a full game instead of just a prologue you can do in 5 minutes would be a different type of endeavor...</span></span></div>
<br /><br />
<br />
<br />JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com1tag:blogger.com,1999:blog-37404288.post-53439036005419401682018-12-04T18:23:00.000+01:002018-12-04T18:33:04.117+01:00Fun with NPM and ExpoSome time ago, I wrote a little game using <a href="https://facebook.github.io/react-native/">React-Native</a> so I could play around in Javascript with React, Mobile development and <a href="https://redux.js.org/">Redux</a>. Yesterday I checked this project again on <a href="https://github.com/JPMoresmau/anthea-quest-rpg">Github</a>, and saw that there was a security issue Github detected and warned me about: the "merge" library had a vulnerability, reported in <a href="https://nvd.nist.gov/vuln/detail/CVE-2018-16469">CVE-2018-16469</a>. OK, time for action!<br />
Of course, my project doesn't reference merge directly. It's a transitive dependency of <a href="https://jestjs.io/">jest</a>. A quick check on the web showed that since I created the project, a lot of things have happened, so my project dependencies are lagging behind. So hopefully upgrading should solve my security issue.<br />
I find instructions at <a href="https://stackoverflow.com/questions/16073603/how-do-i-update-each-dependency-in-package-json-to-the-latest-version">https://stackoverflow.com/questions/16073603/how-do-i-update-each-dependency-in-package-json-to-the-latest-version</a>, so I run <span style="font-family: "courier new" , "courier" , monospace;">npm outdated</span>, which indeed confirms a lot of my dependencies are old.<br />
I install <a href="https://www.npmjs.com/package/npm-check-updates">npm-check-updates</a>:<br />
<span style="font-family: "courier new" , "courier" , monospace;">npm i -g npm-check-updates</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">ncu -u</span><br />
As indicated, and I get:<br />
<span style="font-family: "courier new" , "courier" , monospace;">Hmmmmm... this is taking a long time. Your console is telling me to wait for input </span><span style="font-family: "courier new" , "courier" , monospace;">on stdin, but maybe that is not what you want.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Try specifying a package file explicitly with --packageFile package.json.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">See <a href="https://github.com/tjunnone/npm-check-updates/issues/136#issuecomment-155721102">https://github.com/tjunnone/npm-check-updates/issues/136#issuecomment-155721102</a></span><br />
<br />
OK, then:<br />
<span style="font-family: "courier new" , "courier" , monospace;">ncu -u --packageFile package.json</span><br />
<br />
This updates everything, and then my app doesn't start anymore. Turns out a lot more changes have happened when I wasn't looking, and the <a href="https://github.com/react-community/create-react-native-app">react native starter app</a> I created has now changed to use <a href="https://github.com/expo/expo-cli">expo-cli</a>. I find <a href="https://blog.expo.io/expo-cli-2-0-released-a7a9c250e99c#de37">some instructions on how to change the npm scripts</a>, but it doesn't solve everthing, I still get an error in the react-native-scripts main file, because the main entry also needs to be changed (thanks <a href="https://docs.expo.io/versions/latest/sdk/register-root-component#i-created-my-project-before-sdk-18">https://docs.expo.io/versions/latest/sdk/register-root-component#i-created-my-project-before-sdk-18</a>). Then it stills fails because the babel module has been renamed, thanks to <a href="https://forums.expo.io/t/problem-upgrading-to-sdk-31-cannot-find-babel-plugin-transform-react-jsx-source/16246">https://forums.expo.io/t/problem-upgrading-to-sdk-31-cannot-find-babel-plugin-transform-react-jsx-source/16246</a>. And then on my expo client I get a Javascript mismatch error, that <a href="https://stackoverflow.com/questions/47763824/react-native-version-mismatch">https://stackoverflow.com/questions/47763824/react-native-version-mismatch</a> helped me solve...<br />
<br />
OK, now my game starts again, I suppose I need to test it thoroughly because something may again decide to break... I did enjoy developing the game, but certainly managing the NPM vulnerabilities and the Javascript ecosystem pace of change can be challenging. I know this isn't an original thought, but experiencing it first hand is definitely an eye-opener!<br />
<br />
<br />JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com1tag:blogger.com,1999:blog-37404288.post-52229298838779356382018-12-03T20:03:00.001+01:002018-12-04T18:24:36.756+01:00Reboot<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Whoa, nearly two
years since I last put something on this blog. I lost the habit, and since I'm
not involved in EclipseFP anymore I don't have the stream of <a href="https://www.haskell.org/">Haskell </a>news to
report I used to. I will now try to reboot it and try to find interesting
experiences to share!</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I was never a big
fan of social media sites, but I use to link to some of these blog posts on
Google +, which is now dying. So this should stay a simple blog, where I have
nothing to sell you and no ads to pester you with!</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Over the last few
months I've built a toy project in <a href="https://facebook.github.io/react-native/">React Native</a>, started learning <a href="https://golang.org/">Go</a>, and done
a lot of things in my day job, so hopefully some interesting stories will
appear! Stay tuned!</div>
<br />JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com0tag:blogger.com,1999:blog-37404288.post-60328115284115437472017-01-21T12:05:00.000+01:002017-01-21T12:05:39.404+01:00So long Haskell, and thanks for all the functional fishI've realized I haven't written or read a line of Haskell in the past 6 months. After roughly ten years of tinkering with it, it seems that I've given up on it. There is a big obvious reason, and other smaller ones.<br />
<br />
Professionally, my job got a lot more challenging (I'm officially an **architect** now, which means I still write lots of code but I have to draw pretty diagrams too (-: ) and involves a lot of research and learning new stuff, things like <a href="https://en.wikipedia.org/wiki/Microservices">microservices</a>, <a href="https://www.docker.com/">docker</a>, <a href="https://www.rabbitmq.com/">messaging</a>, <a href="https://angular.io/">Angular2</a>, <a href="https://cordova.apache.org/">mobile apps</a>, etc. So a lot of my time is dedicated to work or to learning for work, so I don't have the time to play around with something quite unrelated like Haskell.<br />
<br />
I suppose to be honest there also was a bit of lassitude with Haskell. I got tired of the less than optimal IDEs, I realized Haskell was not going to get me a great job, and there were a few little skirmishes on the web that got ugly and made me see the community in a less favorable light.<br />
<br />
This was fun, though, and I certainly learned a lot, and I hope it has made me a better programmer. A lot of my coding now is done in <a href="https://dzone.com/articles/functional-programming-java-8">Java 8</a>, and it's good to be able to apply some functional idioms practiced in Haskell, and more general ideas like data immutability, small pure functions do help make better - more testable, easier to understand - code.<br />
<br />
So maybe I'll come back to Haskell some day, but not for now. To all the open source projects I've contributed, I wish you the best!<br />
<br />
Happy Haskell (or any other language) Hacking!<br />
<br />
<br />JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com2tag:blogger.com,1999:blog-37404288.post-41596655626901144552016-10-01T15:09:00.000+02:002016-10-01T15:09:24.134+02:00Everything is broken<div style="line-height: 100%; margin-bottom: 0in;">
This week was I
suppose fairly typical. Started using a new library, the excellent
<a href="http://umlg.org/sqlg.html">sqlg</a> that provides the <a href="http://tinkerpop.apache.org/">TinkerPop</a> graph API on top of relational
databases. Found a bug pretty quickly. Off we go to contribute to
another open source project, good for my street cred I suppose. Let’s
fork it, and open the source code in <a href="https://www.jetbrains.com/idea/download">IDEA</a> (Community edition). After
years of hearing abuse about Eclipse, I’m now trying to use “the
best IDE ever” (say all the fan boys) instead. Well, that didn’t
go so well, apparently importing a Maven project and resolving the
dependencies proves too much for IDEA. I fought with it for a while,
then gave up.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
Fired up Eclipse, it
opened and built the sqlg projects without a hitch. Wrote a test,
fixed the bug, raised a <a href="https://github.com/pietermartin/sqlg/pull/75">PR</a>, got it accepted with a thank you, life is
good.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
Then I find another
bug. Except that upon investigation, it’s not in sqlg, it’s in
the actual <a href="https://issues.apache.org/jira/browse/TINKERPOP-1483">TinkerPop code</a>. The generics on a map are wrong, there are
values that are not instances of the key class (thanks generics type
erasure!). So I can fix by changing the method signature, or change
the keys. Both will break existing code. Sigh…</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
Oh, and the
TinkerPop project doesn’t build in Eclipse. The Eclipse compiler
chokes on some Java 8 code. Off to the <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=502871">Eclipse bug tracker.</a> Maybe I
need to have three different Java IDEs to be able to handle all the
projects I may find bugs in.</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br />
</div>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<a href="http://bobdylan.com/songs/everything-broken/">Everything isbroken</a>. Off I go to my own code to add my own bugs.</div>
JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com1tag:blogger.com,1999:blog-37404288.post-43966948283880391642016-08-03T20:26:00.001+02:002016-08-03T20:26:59.232+02:00Scratching an itch: generating Cabal data-files field automaticallyMaybe I didn't look hard enough, but I'm not aware of a tool to generate the contents of the <a href="https://www.haskell.org/cabal/users-guide/developing-packages.html#package-properties">Cabal data-files field</a> automatically when you have loads of folders to include. Cabal has very simple wildcard matching for files references in this field, by design (to avoid including too much data in a source distribution). So it only supports wildcards to replace the file name inside a directory for a given extension, and doesn't support sub directories.<br />
<br />
For the reload project - first release on <a href="http://hackage.haskell.org/package/reload">Hackage</a>! - I had to include <a href="https://github.com/JPMoresmau/reload/blob/master/reload.cabal">loads of files</a>, all the Polymer web components the UI depends on, which are all on different sub directories, with a bunch of different extensions. So I wrote <a href="https://github.com/JPMoresmau/data-files-gen">a little tool</a> to generate the field automatically, and put it on <a href="http://hackage.haskell.org/package/data-files-gen">Hackage</a> too.<br />
<br />
You pass it a directory name, possibly some subdirectories and extensions to ignore, and it generates all the required entries. Saved me loads of time, and scratched my own itch!JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com1tag:blogger.com,1999:blog-37404288.post-12733432488095364032016-07-16T16:16:00.000+02:002016-07-16T16:16:29.577+02:00Another Web-based Haskell IDEAfter giving up on EclipseFP, I've worked a bit on <a href="https://github.com/haskell/haskell-ide-engine">haskell-ide-engine</a> and <a href="http://leksah.org/">leksah</a>, contributing little things here and there to try to make the Haskell IDE ecosystem a little bit better. But at some point, I tried to update the GTK libraries on my Ubuntu machine to get leksah to run, and broke my whole desktop. Hours of fun followed to get back to a working system. So I thought again at my efforts last year to have a web based IDE for Haskell, because using the browser as the UI saves users a lot of pain, no UI libraries to install or update!<br />
<br />
I started another little effort that I call "reload", both because it's another take on something I had started before and of course because it issues ":reload" commands to ghci when you change files. I have changes the setup, though. Now I use <a href="https://hackage.haskell.org/package/scotty">Scotty</a> for the back end, with a REST API, and I use a pure Javascript front-end, with the <a href="https://www.polymer-project.org/1.0/">Polymer</a> library providing the web component framework and material design. I also use a web socket to send back GHCi results from the back end to the browser. I still use <a href="https://github.com/ndmitchell/ghcid">ghcid</a> for the backend, maybe one day when haskell-ide-engine is released I can use that instead.<br />
<br />
The functionality is fairly simple yet: there is a file browser on the left, and the editor (I'm using the <a href="https://ace.c9.io/#nav=about">ACE</a> web editor) on the right. There is no save button, any change is automatically saved to disk (you use source version control, right?). On the server, there is a GHCi session for each cabal component in your project, and any change causes a reload, and you can see the errors/warnings in a menu and in the editor's annotations. You can build, run tests and benchmarks, and I've just added ":info" support. The fact that we're using GHCi makes it fast, but I'm sure there's loads of wrinkles to iron out still.<br />
<br />
Anyway, if you're interested in a test ride, just clone from <a href="https://github.com/JPMoresmau/reload">Github</a> and get going!JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com3tag:blogger.com,1999:blog-37404288.post-89179866809072314642016-02-23T19:00:00.000+01:002016-02-23T19:00:04.677+01:00Haskell Through of DisillusionmentI'm going through a hard pass with <a href="https://www.haskell.org/">Haskell</a>. I still love the language, of course, but some things in the ecosystem bother me as they impact seriously both the fun of writing Haskell and my productivity.<br />
<br />
Sometimes it's the lack of good development environment that gets me. I have failed with EclipseFP to build a community and gather enough support, but it doesn't seem that other efforts go that much further. I contribute to <a href="http://leksah.org/">Leksah</a> and <a href="https://github.com/haskell/haskell-ide-engine">haskell-ide-engine</a>, and there a plugins now for <a href="https://atom.io/packages/ide-haskell">Atom</a> or other modern editors, but when I do a spot of <a href="http://developer.android.com/guide/index.html">Android</a> development I see what a good <a href="http://developer.android.com/sdk/index.html">IDE</a> is and how much I miss in Haskell.<br />
<br />
But today it's more the open source libraries issues that irks me. It's great that we have loads of libraries, and they're open source and usually good quality. But of course the maintainers are all volunteers, and sometimes have better things to do. But there are a few libraries that I use in my code that now actually stop me from progressing. I have provided enhancements or bug fixes that I need for my projects as pull requests, and they languish in the maintainers' inboxes for months. So what am I to do? Hound the maintainers? Fork the library to apply my patches? Rewrite my code so it doesn't use that library but another, better maintained? Not use libraries but write everything myself? And of course if I offer to take over maintainership I'll end up being overloaded and will perpetuate the problem. I suppose the best approach will be to offer to be one of MANY maintainers for the library, so that I can merge my changes and release on Hackage if the others maintainers are otherwise busy/uninterested. I'm not sure how that can work in the general case, though, if loads of people are maintainers for loads of libraries, I believe that having one person with the vision and the drive for a project is best, but for little libraries it may not matter much.<br />
<br />
<br />JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com3tag:blogger.com,1999:blog-37404288.post-51578225317879514722016-02-11T17:56:00.000+01:002016-02-11T17:56:22.551+01:00Starting out on AndroidOohhh, it's been a while since I've posted. I spent some time working on <a href="https://github.com/haskell/haskell-ide-engine">haskell-ide-engine</a>, but then I got Haskell fatigue and decided to look at Android instead. A change is as good as a rest!<br />
<br />
Coming from the Java world, it's not difficult to get into Android. I followed the <a href="http://developer.android.com/guide/index.html">guide</a> on the Google site to get the basics. At first I got a bit afraid of the unholy mix of Gradle build scripts, visual layout editor, XML files and Java code, but you get over it. The IDE has some nice touches and it's good to get useful autocompletion in resources like ids and strings. I liked the warning about SDK level APIs and missing resources once I started translated my app into French.<br />
<br />
But I have to say, when I was working on EclipseFP I often ran into IDEA fanatics that swore that IDEA was miles ahead of Eclipse. Android Studio famously <a href="https://jaxenter.com/android-studio-dumps-eclipse-for-intellij-idea-106065.html">moved</a> from Eclipse to IDEA, and frankly, I don't see what the hype is about. Yes, Eclipse has some annoying bugs and idiosyncrasies. But Android Studio, at least with the default settings, as I haven't spent much time customizing it, is not such a wonderful IDE. It's slow to start up, the view layout is sometimes confusing, the font is too fine or small in places (I must be getting old, but sometimes I couldn't see that a semi colon was in fact a colon) and the warning/error markers in the gutter are way too small and nearly invisible, to the point that sometimes the build fails and I can't see where in the source is the error! Maybe there's a special "grumpy old man with failing eye sight" setting to make things bigger.<br />
<br />
Since I believe that you learn by doing, I've developed my first app, and I'm in the process of publishing it to the App Store (so I can claim I'm a published Android developer in my CV, wink). Apart from the fact that you need to supply an icon in 2 million different sizes, which is challenging for somebody with my artistic abilities, it's a straightforward process. My app is an uninteresting workout log app (to record how much weight you supposedly lift at the gym) that's purely local (no social features, so you can't boast online about the weights you may have lifted), but hey, it's a start. The code is of course on <a href="https://github.com/JPMoresmau/JPWorkoutLog">Github</a>.<br />
<br />
I've also wrote a basic game using the framework presented <a href="http://www.kilobolt.com/game-development-tutorial.html">here</a>, and the nice asset from <a href="http://kenney.nl/assets">Kenney</a> (see note about artistic abilities above). Nothing fancy, just a rabbit hopping over holes and getting carrots for points, but just seeing my little game running on my family's phones and tablets is nice! It's on <a href="https://github.com/JPMoresmau/RabbitPlatform">Github</a> too.JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com1tag:blogger.com,1999:blog-37404288.post-90263892479823471452015-09-30T22:10:00.000+02:002015-09-30T22:10:04.279+02:00Symbolic differentiation to the rescue! ... Or not.I'm still playing with my <a href="https://en.wikipedia.org/wiki/Long_short-term_memory">LSTM</a> networks, inspired by a few blog posts about their "<a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">unreasonable efficiency</a>". I spent a lot of time messing with genetic algorithms, but then I came back to more predictable methods, namely <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent">gradient descent</a>. I was trying to optimize the performance of the cost function I use via <a href="http://hackage.haskell.org/package/ad">AD</a> (even asking help on <a href="http://stackoverflow.com/questions/32765313/optimize-a-list-function-that-creates-too-much-garbage-not-stack-overflow">stack overflow</a>) when I stumbled across a couple of blog posts on symbolic differentiation (<a href="http://5outh.blogspot.in/2013/05/symbolic-calculus-in-haskell.html">here</a> and <a href="http://jtobin.ca/blog/2014/07/06/automasymbolic-differentiation/">here</a>). The last one, combining automatic and symbolic differentiation, struck a chord. If my differentiation calculation was taking so much time to calculate, could I just not calculate it once with symbolic expressions, then close the resulting expression over my variables (my LSTM network) repeatedly while applying the gradients. I should only pay the price for the derivation once!<br />
<br />
So I extended the data type suggested in the blog post to include all operations I was using in my function, manage to sort out all the types and verify <a href="https://github.com/JPMoresmau/rnn/blob/master/test/rnn-test.hs#L324">via a few tests</a> that it would work. I had great hopes! But as soon as I started testing on a real LSTM, the code just crawled to a halt. I even thought I had some infinite loop, maybe some recursion on the expression, but testing more thoroughly showed that it was the sheer size of the generated expression that was the issue. A LSTM of 2 cells is represented in the cost function used by AD as an array of 44 doubles, so basically for a LSTM of 2 cells, I'll have 44 variables in my gradient expression. My simple test that tries to use a LSTM to generate the string "hello world!" uses 9 cells (9 different characters in the string) , which is 702 variables. Even printing the expression takes forever. Running it through a simplifying step takes also forever. So my idea was not as good as it first looked, but it was fun testing it!<br />
<br />
The code for my expressions can be found <a href="https://github.com/JPMoresmau/rnn/blob/master/src/AI/Network/RNN/Expr.hs">here</a>, and the code doing the gradient descent via the symbolic differentiation is <a href="https://github.com/JPMoresmau/rnn/blob/master/src/AI/Network/RNN/LSTM.hs#L211">here</a>. All of that looks probably very naive for you calculus and machine learning experts but hey, I'm a human learning... If everybody has any idea to speed up my code, I'l happily listen!JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com2tag:blogger.com,1999:blog-37404288.post-10650719599676508652015-08-28T17:41:00.000+02:002015-08-28T17:41:00.415+02:00A Reddit clone (very basic) using Local Storage and no server side storageSome weeks ago there was a bit of a tussle over at Reddit, with subreddits going private in protest, talk of censorship, etc. This was interesting to see, from a distance. It got me thinking about trying to replicate Reddit, a site where people can share stuff and have discussions, but without having the server control all the data. So I've developed a very basic Reddit clone, where you can post links and stories and comment on them. You can also upvote stories and comments, and downvote what you upvoted (cancel your upvote). But there's a catch: the site has no database. Things are kept in memory, and on the users machine, via the HTML 5 <a href="http://diveintohtml5.info/storage.html">LocalStorage</a>. That's all!<br />
<br />
Everytime you upload or upvote something, it gets saved to your LocalStorage for the site. Once something gets downvoted to zero, it disappear. When you go to the site, whatever is in your LocalStorage gets uploaded and upvoted again. So stories can come and go as users connect and disconnect, and only the most popular stories will always be visible on the site (since at least one connected user needs to have uploaded or upvoted a story for it to be visible).<br />
<br />
Of course, there is still a server, that could decide to censor stories, modify text, but at least you can always check that what you have on YOUR machine is the data you wanted. you can always copy that data elsewhere easily for safe keeping (browser developer tools let you inspect your LocalStorage content).<br />
<br />
All in all, this was probably only an excuse for me to play with Javascript and Java (I did the server side in Java, since it was both easy to build and deploy) and Heroku. I've deployed the app at <a href="https://fivemegs.herokuapp.com/">https://fivemegs.herokuapp.com </a> and the source code can be found at <a href="https://github.com/JPMoresmau/5megs">https://github.com/JPMoresmau/5megs</a>. Any feedback welcome!JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com1tag:blogger.com,1999:blog-37404288.post-9243629287018404522015-08-02T15:44:00.003+02:002015-08-02T15:46:54.669+02:00Playing with Recurrent Neural Networks in HaskellSome time ago an <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">interesting article</a> surfaced on Reddit, about using recurrent neural networks to generate plausible looking text. <a href="http://karpathy.github.io/">Andrej</a> did a very good job in explaining how they work and some of the techniques and algorithms he used. I thought this was worth some explorations on my own, so I did a bit of research and tried to implement my own networks in <a href="https://wiki.haskell.org/AI">Haskell</a>.<br />
<br />
I implemented the basics using the <a href="http://hackage.haskell.org/package/hmatrix">hmatrix</a> package to have vectors and matrices. I went to <a href="https://www.khanacademy.org/math/precalculus/precalc-matrices">Khan Academy</a> to learn more about these topics because I had stopped math in school before learning matrices. Once I managed to implement hopefully correctly the algorithm I used the simple-genetic-algorithm package (well, <a href="https://github.com/JPMoresmau/simple-genetic-algorithm-mr">my fork</a>, uploaded on <a href="http://hackage.haskell.org/package/simple-genetic-algorithm-mr">hackage</a>) to evolve the network via <a href="https://en.wikipedia.org/wiki/Genetic_algorithm">genetic selection and mutation</a>.<br />
<br />
This gave some results, especially when I fixed my <a href="https://en.wikipedia.org/wiki/Mutation_(genetic_algorithm)">mutation</a> code, that in fact was doing nothing, thus emphasizing again that mutation is a critical part of genetic algorithms, and not just crossovers.<br />
<br />
Then I went to implement proper learning algorithms, a bit less random than genetic evolution. I was clearly out of my depth there, but learned a lot. To implement gradient descent I actually used the <a href="http://hackage.haskell.org/package/ad">ad</a> library and had to rewrite the algorithms without hmatrix so they would work on simple lists. Given that a couple of weeks again I didn't even understand what <a href="https://en.wikipedia.org/wiki/Automatic_differentiation">AD</a> was, I'm happy I got some stuff to work. I was lucky to find some <a href="http://deeplearning.net/tutorial/lstm.html">good code in python</a>, even trying to understand the under-specified <a href="http://climin.readthedocs.org/en/latest/rmsprop.html">RMSProp</a> algorithm.<br />
<br />
The end result is not great, though. My network can learn the really complex sentence "Hello world!" and regenerate it after a couple of thousand iterations of the learning algorithm, in around a minute on my machine. I haven't managed to parallelize the treatment yet, and running on all my cores instead of just one make performance a lot worse. On more complex sentences performance becomes really bad, as both the network becomes bigger (more characters to deal with, etc) and the data to evaluate is bigger.<br />
<br />
Trying to use a sparse representation for characters using 2 values instead of 1 as in the standard <a href="https://en.wikipedia.org/wiki/Categorical_distribution">1-of-k</a> encoding didn't work well. I also realize that tweaks in the cost function had a huge impact on how well and how fast the network can learn. I have started to look at cutting the data in batches but I think I need to make the learning on simple data as fast as possible before moving on.<br />
<br />
So, nothing earth shattering, no great progress or insight, but I feel I learned a lot by implementing my own stuff and running into all the issues that are probably well known of people familiar with these subjects. My artificial intelligence is still pretty dumb, but hopefully my own natural intelligence has progressed a bit!<br />
<br />
The code is on <a href="https://github.com/JPMoresmau/rnn">github</a> of course, so if anybody wants to have a look, I'll gladly take any feedback!! JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.com2