Ever since I upgraded to Wordpress 2.5 I had to moderate every comment posted on this blog. Note that I hadn't turned this on in the Wordpress settings.
Some research showed that it might have to do something with 'older' plugins. After a trail and error period I found out that the WP Hashcash plugin was to blame. Ever since I de-activated it, the problem seems to be gone.