tag:blogger.com,1999:blog-37404288.post9026389247982347145..comments2023-11-02T14:40:18.756+01:00Comments on JP Moresmau's Programming Blog: Symbolic differentiation to the rescue! ... Or not.JP Moresmauhttp://www.blogger.com/profile/09964251063221757176noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-37404288.post-28115436601986211432017-10-02T10:26:38.004+02:002017-10-02T10:26:38.004+02:00This comment has been removed by the author.harada57https://www.blogger.com/profile/06015023155124017905noreply@blogger.comtag:blogger.com,1999:blog-37404288.post-10138821385105450912016-07-23T19:02:14.802+02:002016-07-23T19:02:14.802+02:00I think your implementation is too slow because of...I think your implementation is too slow because of AD package.<br /><br />I made simple brute-force implementation of RNN in haskell from scratch and it works pretty fast - two-layer GRURNN with 32 units in each layer steps forward/backward pass in 20ms for input lines of ~100 characters.<br /><br />I tried to arrange things in a way which allows unit code to look as much similar as possible to math description found in articles, so vanilla-RNN unit code looks like<br /><br /> expr <- store s h (relu (whx' * inp + whh'*h' + b'))<br /><br />and GRU unit code looks like<br /><br /> let rt = sigm (wrx' * inp + wrh' * h' + br')<br /> zt = sigm (wzx' * inp + wzh' * h' + bz')<br /> ht = tanh (whx' * inp + whh' * (rt * h') + bh')<br /> gruExpr <- store s _h (zt * ht + (1.0 - zt) * ht)<br /><br />Maybe, I'll write blog post about my implementation soon.Anonymoushttps://www.blogger.com/profile/00472122942963715738noreply@blogger.com