peteg's blog 2007-12-06-HOPE-Unicode.autumn

Haskell server-side integration for FCKeditor.

I am not a JavaScript hacker, so I have no clear idea how best to use FCKeditor. My embryonic Haskell library just spits out either a textarea or some JavaScript that creates an FCKeditor instance depending on how HTTP_USER_AGENT is set, though I can imagine someone wanting to do something fancier [*]. The POSTed data is validated against XHTML 1.0 Strict using HaXml, which seems to work well for the most part; for some reason FCKeditor uses the non-standard <embed> tag for Flash content, and I can't find a convincing reason why [**].

In the not-to-distant future I will implement the connector stuff, and Cabalise it.

[*] Apparently I still need to crank out an <iframe> to satisfy Internet Explorer, so we can either revert to XHTML 1.0 Transitional or generate some non-standard XHTML just for Internet Explorer. It's a tough call.

[**] It seems that recent versions of Internet Explorer (6 and 7), Mozilla-based browsers (Camino, FireFox) and Safari 3 are all happy with the <object> tag. Adobe has a "knowledge base" article full of non-reasons to use the <embed> tag. The great thing about web standards is we're all empiricists now...

Lest I forget, Haskell and Unicode.

One reason I ran away from all of the CMS systems implemented in PHP is its (historically) crappy support for Unicode [*]. Standard Haskell, on the other hand, has required the Char type to be able to represent a Unicode codepoint for quite a while now. Unfortunately there are a few libraries that are not Unicode friendly, such as just about every library interfacing with C.

Concretely:

HSQL needed some work to get it to talk UTF-8 to PostgreSQL.
Most but not all of the CGI library is Unicode friendly. I don't know enough about the various RFCs to know what's encoded as what, so I don't know how to do this right. For example, how are Unicode filenames handled?
The regexp libs are a bit of a minefield (the user-interface is quite complex, and those C libraries are unknown quantities), so I have avoided using them.
HOPE itself is almost entirely encoding-agnostic, apart from the top-level (where it builds a CGI header for the webserver's consumption), and HaskellDB just punts around the strings fairly blindly, doing a minimal amount of escaping. Good job, Björn.

I really, really wish Haskell had a decent story about character encoding at the I/O level. Back in 2002 people seemed to get really excited about doing something about it, but that mailing list is dead now. I guess the hope is that once ByteStrings and all that are bedded down, the I/O layer can be rebuilt on efficient foundations, fusion will take care of performance issues with codec layers and so forth.

Update: ConradP has surveyed some Haskell character munging libraries.

[*] perl has good Unicode support, if one is happy to play the guessing game as to what format each string is in. I feel that strong typing — clearly separating characters from strings of bytes — is just what is needed here.

Halfway through the project, I begin to talk about the project.

So the game here is to build a CMS-style website for DRD, who are presently using an unmaintainable ASP mess. (Heh, I think that's the old ASP, not ASP.NET, but what would I know.) I decided to renovate Björn's Haskell effort, HOPE, which looked, superficially at least, pretty hackable.

Activity for these past few months:

I tried to fix the concurrency issues. There was/is [*] a lot of confusing code that looks like it might be safe, but wasn't. It might have worked if the DBMS provides coarse enough concurrency, and traffic is sufficiently light. (I don't claim to have fixed everything yet, and there are limits to what we can do.)
As part of the above I hacked the daylights out of HaskellDB and HSQL, but only conforming their PostgreSQL backends with my higher-level changes [**]. Specifically I tried to extend their notions of a relational database to encompass constraints [***], and add support for the serial datatype.
- HSQL seems adequate as a low-level SQL interface, at least as far as these things go in Haskell [***], so I don't know why anyone would reinvent that wheel (ask them).
- I would strongly recommend against trying to use HaskellDB, despite the heroic efforts of Björn et al. It's nice in theory but quite limited and very complex in practice. If I were to do this project over, I would drop HOPE's dependency on HaskellDB.
- I am now painfully aware of the semantic gap between Haskell and SQL databases. What we really want is serialisation and querying of algebraic data types, that is, something closer to XML technology. The only group I know that is taking persistence seriously at the typed, higher-order, etc. programming language level is the mob working on Alice/ML, and if I had a spare life I'd marry that with Benjamin C. Pierce's work of the past ten years or so and develop a mergeable, distributed, queryable storage manager for a decent language.
Added a lot of I18N support. This is as-yet incomplete, of course, and I'm not very happy with how I've done the dynamic part of it. One major outstanding issue is how best to support multi-lingual tagging.
Shifted away from Björn's home-brew and somewhat buggy hmarkup to the Windows-user friendly FCKeditor. I have my qualms about this, but I've got to consider my user-base.

Some of the abstractions in HOPE are fantastic, and others are head-scratching, tantalisingly close to being so. If I have the time and enough brain capacity, I'd really like to re-do the notion of resource so we can (for example) generate site maps and have fewer URL paths scattered through the code. So, good effort Björn.

If you're interested in any of this, you can take a look at the darcs repos at http://peteg.org/haskell. Please note that everything there should be considered alpha quality and under chaotic development.

[*] My changes are so pervasive that it's better to think of my version as a fork rather than a continuation. The database schema is quite different and currently requires PostgreSQL, so I doubt it is useful to any current users.

[**] This has some nasty ramifications. One is that it is unlikely that my code will be merged into the mainstream darcs repos, as I have no interest in or time to fix the other backends. (I refuse to encourage anyone to use speed-over-correctness software like MySQL.) Due to this, I doubt one can use the shiny-new cabal-install to suck down the myriad dependencies of my version of HOPE, as you'll need some stuff from my repos, and other stuff may as well come from the official places.

[***] Somewhat ironic to me is that all the low-level Haskell SQL bridges I've seen have a very limited view of what a relational database is; usually the bridge just ships SQL one way and gets a list of rows back, and provides a very basic table description mechanism. I haven't seen any support for defaults, triggers, constraints (foreign keys, primary keys, uniqueness, etc.), and while there is usually support for transactions, it is difficult to figure out what that means as the bridges all try to be backend-agnostic. Conversely there are a lot of attempts at making rows and queries type-safe.