First assets for the Valley of Madness

Meet the butterfly and happy cloud (shooting rainbows) in Harry Dies. World 2 will be a lovely valley full of waterfalls and death. Killer butterfly, exploding flowers you name it! Here are the first two animated enemies I made.

game enemy butterflygame enemy happy cloudbullet rainbow

Background objects will include waterfalls, trees and clouds (not shooting stuff fortunately... but why not?).

background objects

Harry Dies

The game

Harry Dies is a super relaxing 2D platform game. Actually that's a lie, it's really a torture platformer where you play Harry the Hairball, save the dragon, smash the princess and hopefully do NOT die (too often).

Oh don't worry, the screenshots won't spoil too many death traps. There are plenty more of them!

Chapter 1: The Abyss of Death

The first level will take you deep into the ocean. You probably should avoid getting sliced, chopped, zapped, eaten… or drown somewhere in the depths.

Harry Dies screenshotHarry Dies screenshot

Chapter 2, 3 and the final fight will follow in further releases. If you want to be notified when this happens, you can follow me on Twitter or my Google+ games collection. Updates will be posted there. Or, if you prefer oldskool, add my RSS feed. The release dates will always be the same: when it's done! If the whole game is finished there will probably be a non-web version for Windows and Android.

If you want to support development, you can use Paypal.


Play Harry Dies right here. Should run on a toaster as long as it supports WebGL.

A keyboard is highly recommended, WASD+SPACE or arrow keys. But there are also touch controls if you're on a tablet. I bow to you if you finish it on a tablet!

The web version will have an ad on top of the screen. Yeah, yeah, a neccessary evil. I will try to configure it to show gaming related stuff.

17.04 Chili

Finally! Sunny days. What looked like a perfect chili season with almost 20°C at the beginning of march, turned out to be a rainy, even snowy, mess until end of April.

But now let battle commence with the following peppers:

  • Trinidad Scorpion: really wanted this one again.
  • Habanero Red: no experiments there, just a good old classic.
  • Fatalii Yellow: best chili I've ever had. Not too deadly. I'd call them perfect.
  • Ghost Pepper: another classic. Jolokia Yellow but 3rd generation. Let's see what we have here.
  • Cayenne Golden: another 2nd generation chili. But since this one already has some fruits and they look like Cayennes are supposed to look, probably no surprises here.
  • Scotch Bonnet: used seeds from a chili pepper I bought somewhere at the city market.
  • Vulcano: now despite the lovely name, this one is actually only a class 3 chili. Rather large fruits if I remember correctly.
  • Lemon Drop: another new one. I bought some fresh lemon drop chilis from Chilifood last year and they were extremly delicious. They indeed tasted a bit like lemon and were not too hot. Probably a good 7 on the scale. These are growing quite large it seems. Almost triple the size of the others. Let's hope they produce pain at that rate, too!

With the weather getting better now, the chilis can now be outside. Right now it's perfectly sunny. Baking in the sun and converting solar energy into pain!

Snø and Cyklus

Our first two Android apps are now on the app store. Interesting experience to develop something from zero to app store without anyone knowing everything better. And it's so much easier to know everything better if you are not involved in the whole thing!

Snø is a weather app with pictures from all around the world. It's probably a bit hard to market, because it fits in neither category. Not a full blown weather app and not a full blown social media picture sharing thingy. Of course it also shows the lunar phases! Apparently everything I do includes that functionality.

And Kianga made Cyklus, a 24-hour watch face with weather forecast. It shares the weather data backend with Snø. Will be interesting to see which one will generate more queries... and costs us more money. Probably Snø in the long run since it's free.

So download it and make the cat happy!

Jessie exim4 spf madness

Here's a strange thing. Seems like spf-tools-perl in jessie have some strange bugs. I never noticed, but after the update from wheezy SPF silently stopped working and the log was spammed with Warning: Unexpected error in SPF check. I traced this down to the invocation of /usr/bin/spfquery.mail-spf-perl which would abort with an error: unresolvable name: localhost at /usr/share/perl5/Mail/SPF/ line 225

The solution to all this madness, /etc/resolv.conf had an entry nameserver localhost. Changing that to made the perl script and SPF checking work again. /etc/hosts does list localhost, no idea why this broke.

Chilis for 2016

Chili season 2016 has already started. I planted 15 about a week ago. This year I don't have an "extreme-only" mix, but variety. Of course no chili season can go without Reapers. So for this year we have:

The Bishop's Crown grow extremly fast, and extremly big. They are also a bit fragile due to their size. Let's hope the first storm doesn't blow them away once they're outside. The list is sorted by heat. Complete meltdown on top and more like bell pepper on the bottom. But I'm still mostly looking forward to the Bishop's Crowns. They were extremly tasty without setting your mouth on fire. For some people, this may be a good thing. Probably not for us that grow chilies. :)

Harvest 2015

This year's harvest is probably less than 2014. Bad weather in the spring, less plants and experiments with soil which turned out to be suboptimal. Not all variants seem to like compressed cocos fibre. The stuff can hold a lot of water, but the plants contantly have "wet feet" and it dries out pretty fast. The turfless soil I tried this year is much better. I will use that again next year. Only downside: it's more annoying to transport. I could probably order it at amazon, but that really feels weird.

So, this year's leaderboards:

Type# of plants# of fruitsRank
Ají Dulce2104th
Carolina Reaper2551st
Trinidad Scorpion115th

Profile: Bishop's Crown

When I first sowed the Bishop's Crown seed in June this year I was more expecting it to become an indoor chili for the window sill. But it turns out they are even faster growers than the Fataliis and after only 1.5 months the plant is 75cm high. Not even a small pot prevented it from growing. It just became offended, letting its leaves hanging, and I had to put it into a bigger pot.

After a bit of research it turned out that these plants grow up to 2-3m high. They seem to grow much larger than your standard capsicum chinense plant. It even grew so fast that the trunk is kinda thin and it seems to have trouble getting enough water to the topmost leaves.

Environmental conditions

Hailing from the Carribean they should be accustomed to tropical climate. Interestingly 35°C behind a window glass (greenhouse like) seems to be almost uncomfortable and 40°C in direct sunlight is too much for them. They seem to like it much better in the shade and, in contrast to other chilis I have, need plenty of water. While others are fine with the soil completely dry for a day or two the Bishop's Crowns seem to prefer it mostly wet.

The Test: Carolina Reaper

The first Carolina Reaper chili was ready today. Compared to other red chilis they look more orangish and quite poisonous if you ask me. Ah well, no pain no gain. So I made a traditional chili con carne and spiced it up quite a bit with a finely chopped chili. The conclusion: they burn as hot as they look! I'd say the stories you've read elsewhere are definitely true. This is a weaponized chili pepper. If you enjoy pain or like to see others gasp for air while they try to eat your evil cuisine, give this one a try. :)

Hetzner root server full disk encryption with cryptsetup LUKS

Updated 29 Dec 2015

This howto turns a Hetzner default Debian root server installation into a fully encrypted server with LUKS that can be unlocked remotely with the dropbear ssh server. If the server is rebooted, seized or a KVM switch is plugged into the wrong server, the data cannot be accessed. Likewise, your data will be lost forever* if you forget the passphrase. This works with Debian Jessie and Wheezy images. But read the note on Jessie below.

(* a long time)

Please note the date of this post. If it is older than a year, the defaults of the Hetzner install image might have changed. Paths and devices might be named differently. I assume the following (Hetzner default) disk layout. If you have configured your server differently, change the device names accordingly.

md0: Raid1 consisting of sda1, sdb1 as swap md1: Raid1 consisting of sda2, sdb2 as /boot md2: Raid1 consisting of sda3, sdb3 as the root filesystem

The server will not be reinstalled from scratch. All data is preserved (read below). You can encrypt an existing server without any config changes to the services it provides.

Debian Jessie

If you're using Jessie and onward, the init system will be systemd by default. Because of an unresolved bug with crypto luks + systemd + certain crypttab options, including keyscript, you'll have two options with this constellation. There are, of course, other more hackish approaches, but it is up to you to evaluate their safety.

Without systemd

Install the package sysvinit-core before you reboot. This removes systemd-sysv, which makes the system default to systemd as PID 1. Your system will continue to use the old sysvinit. Nothing else will change. Everything that worked in Wheezy will continue to do so.

With systemd

First copy /sbin/init to /sbin/init.sysv This will give an easy way back to sysvinit if thins go haywire. Install systemd-sysv to switch the default to systemd. Add a kernel parameter luks=no to the kernel command line in /etc/default/grub and run update-grub && update-initramfs -u. This will now tell systemd not to unlock the disk at boot, the old scripts in initramfs will take care of that.

There is one important note however: no other disk will be unlocked at boot time, only root. So if you have any other entry in crypttab, it will not be available to auto-mounting at boot and systemd will fail to bring up that mount point. That includes the encrypted swap further down in this guide. You will have to mount the rest of the paritions manually after each boot. If it's just swap, you could also remove it completely, 32GB or even 64GB in the current servers should be enough for almost every workload.

If the system now fails to boot, wait at least 1m30s, that's how long systemd tried to mount the missing devices until it gave up on my test system, pass the following kernel parameter on boot: init=/sbin/init.sysv. You can either do that with a LARA (KVM) console or edit your config in the rescue system. That will switch to sysvinit for this one boot.

Additional packages that must be present on your current system

Install these in your current installation before you proceed.

apt-get install cryptsetup apt-get install busybox dropbear

You should have cryptsetup installed before you install dropbear. There might be some strange side effects with the initramfs config otherwise. I have received such a report. At least it doesn't hurt.


There is a complete description on how to generate the keys and unlock the disk in the cryptsetup documentation. It is installed in /usr/share/doc/cryptsetup/README.remote.gz. Configure dropbear following the readme before you proceed.

If you have configured it and had the private/public SSH keys generated automatically, do not forget to copy the private key or install your public key according to the readme.

Reboot into rescue system

Create a backup if you're working with important data!

Log into the robot and reboot the server into the (64bit) rescue system. At the time of this writing every tool was available in the rescue system.

Save your old system

The old system is moved aside while the encrypted disk is prepared. If you have more data than what fits onto the root filesystem, you need to move large data directories either to the backup FTP or restore them from a backup afterwards. In the rescue system you will have a root filesystem ramdisk which is half the size of the server's memory. So if you have a 32GB root server you will have ~16GB available.

mkdir /oldroot mount /dev/md2 /mnt rsync -a /mnt/ /oldroot/ umount /mnt

Your system is now mirrored into /oldroot.

Create the encrypted disk

Let's create the encrypted filesystem. This will destroy all data on your server! You might want to use a diceware password so you don't have to enter any special characters while booting the system. Example on xkcd. But don't use an online generator or the horse staple. :) Or for example: dd if=/dev/random bs=1 count=18 | openssl base64

The following command will create a LUKS device using the aes-xts-plain64 cipher with 512 bits key size and a number of key iterations that will take roughly 5 seconds to compute on your hardware. Meaning it will take 5 seconds to unlock the disk. Since servers are rarely rebooted, it does not hurt.

cryptsetup --cipher aes-xts-plain64 -s 512 --iter-time 5000 luksFormat /dev/md2

You can verify that everything went fine with the luksDump command. It should produce something like the following output:

# cryptsetup luksDump /dev/md2 LUKS header information for /dev/md2 Version: 1 Cipher name: aes Cipher mode: xts-plain64 Hash spec: sha1 Payload offset: 4096 MK bits: 512 MK digest: de ad be ef MK salt: de ad be ef de ea be ef MK iterations: 83750 UUID: 5f76b99e-0418-4fb5-a731-201004ede948 Key Slot 0: ENABLED Iterations: 336005 Salt: de ea be ef de ea be ef Key material offset: 8 AF stripes: 4000 Key Slot 1: DISABLED Key Slot 2: DISABLED Key Slot 3: DISABLED Key Slot 4: DISABLED Key Slot 5: DISABLED Key Slot 6: DISABLED Key Slot 7: DISABLED

Once the LUKS device is created, activate it. This will ask for the passphrase you've just used.

cryptsetup luksOpen /dev/md2 root

The following step is optional but recommended. It fills the whole disk with random data and overwrites anything on the partition that may be recoverable. Even though it reads from /dev/zero, you are writing to the high level LUKS device which writes random encrypted noise to the disk. This might take a while. 2-4 hours with the current disk sizes. You can skip this step and create a large file on the filesystem after everything is finished with the same effect.

dd if=/dev/zero of=/dev/mapper/root bs=1M

Create a filesystem on the encrypted device. I will use ext4 with default options and set check interval to 6 months, error behavior to remount-ro and max mount count to 50.

mke2fs -t ext4 /dev/mapper/root tune2fs -i 6m -e remount-ro -c 50 /dev/mapper/root

You can now restore your data.

Reinstall the system

mkdir /newroot mount /dev/mapper/root /newroot rsync -a /oldroot/ /newroot/

Before we can changeroot into the new system we need some special file systems and the boot file system inside. These are needed so grub and initramfs reinstallation work.

mount /dev/md1 /newroot/boot mount --bind /dev /newroot/dev mount --bind /sys /newroot/sys mount --bind /proc /newroot/proc chroot /newroot

The fstab needs to be changed to reflect the new root file system. Edit the file and replace / (root) with the following:

/etc/fstab: /dev/mapper/root / ext4 defaults 0 2

To be able to boot from the encrypted file system we need a crypttab.

/etc/crypttab: # <target name> <source device> <key file> <options> root /dev/md2 none luks

Now all is set. We need to tell grub the new root device and regenerate the initramfs to include the encryption stuff. Run the following commands. They should automatically pick up /dev/mapper/root as the new root device.

update-initramfs -u update-grub grub-install /dev/sda grub-install /dev/sdb


IPv6 will not be running after a reboot. The IPv4 networking will be brought up in the preboot ssh server. Once Debian boots the real system, configuring the network interfaces will fail, because eth0 was already configured with ip earlier and the network will be in a somewhat inconsistent state. I fixed this be forcing the network down and up at the end of the boot sequence. Add the following two lines to your /etc/rc.local:

/sbin/ifdown --force eth0 /sbin/ifup --force eth0


You can now leave the chroot, reboot the server and hope for the best. :)

logout umount /newroot/boot umount /newroot/proc umount /newroot/sys umount /newroot/dev umount /newroot sync reboot

The system will now boot into the dropbear ssh server which you can connect to and enter the passphrase to unlock the disk. The system will then continue to boot. Once the server answers pings again, ssh into it. To enter the passphrase use the following command:

echo -n "YOUR SUPER SECRET PASSPHRASE" >/lib/cryptsetup/passfifo

The system will now disconnect your ssh. If not you may have used the wrong passphrase. My terminal was kind of broken afterwards and I had to restart the shell. You now should be able to connect normally.

If you have skipped the optional step (writing the disk full) you should do it now. The easiest way is to create a large file with dd.

dd if=/dev/zero of=foofile bs=1M

You should stop any databases or other software that does not like "No space left on device" before writing the disk full. <Insert Mysql bashing here>

Encrypted swap

The swap can also be encrypted with a random key at boot time. This works automagically. Add the following line into your /etc/crypttab. This sets up an encrypted swap device with a random key as /dev/mapper/swap everytime the system boots.

swap /dev/md0 /dev/urandom swap,cipher=aes-cbc-essiv:sha256

And change the swap entry in your fstab into the following:

/dev/mapper/swap none swap sw 0 0


Don't panic. After all you have backups!

If you cannot log in via dropbear at all, because you have forgotten to copy the generated private key to your system or did not put your public key onto the server, it can be fixed easily. Just boot back into the rescue system. You can edit the config and regenerate the initramfs with the same commands as used above.

If login does work, first check the passphrase. You never know, you might just have a typo. If it still does not work, reboot the system into the rescue mode. Try unlocking the disk manually.

cryptsetup luksOpen /dev/md2 root

If that does not work you might have mistyped it while creating the LUKS device. Either try some common typos or you will have to start over.

If you can successfully unlock the disk, mount it and check if you have a problem with the dropbear configuration. See the readme mentioned in the section "Preparation" for any errors you might have in your configuration.

What has changed now?

The encryption is completely transparent. The only user-visible things that have changed are the following devices:

/dev/md0 is now /dev/mapper/swap /dev/md2 is now /dev/mapper/root

Everything else, including disaster recovery from failed hard drives has not changed. The disks are still part of the underlying MD device and do not know anything about the encryption. The boot partition is still unencrypted.

How secure is it?

The encryption itself should be bullet proof as long as the passphrase is complex enough of course. But even if it is "i hate passwords" you should be fairly safe since we used 5 seconds worth of iterations. On Hetzner's EX40 with an i7-4770 it results in 1.5 million iterations and that is (was? see post date!) a very fast CPU.

The boot partition is unencrypted. If someone gained access to your robot account and boots the server into the rescue system he can, of course, modify the initramfs and install software that will compromise your system if you login. The robot logs these reboot requests so if you didn't request it, be very wary and do not log in. The encrypted data is most likely not compromised or modified. We used aes-xts-plain64 which does not suffer from an issue like the default cbc-essiv mode. See for example the description of the malleability attack.

You could probably make a copy of your boot partition which you can restore in case of an unexplained reboot or checksums of the files which you can compare while in rescue mode.

If you need more security than that, only systems with TPM can provide a secure boot path that prevents tampering.


If you want, you can use the checklist below while you set up the server. All important steps are recorded, so you're unlikely to miss one.

← newer stuff older stuff →

Index: 1 - 2 - 3 - 4 - 5 - 6