tech & sp

How to force a local DNS resolver to be used using resolvconf

2011-06-01T21:53:00.000+02:00

I know it has been a while, but after reading a blog post by Anand Kumria over at planet.debian.org I decided to have a quick look at one of the problems he described.

Basically, Anand wants to force the local resolver to be used for each and every network connection, may that connection be established manually or via NetworkManager. He wrote that fixing this configuration for every new connection manually is tedious, and I fully agree on that. So here is a solution to do this all automatically, using resolvconf:

After installing the resolvconf package every time /etc/resolv.conf is to be updated resolvconf takes care of that. Using the files in /etc/resolvconf this process can be controlled and the resulting file modified to fit one own's needs.

So at first we would like the local resolver to be used for every connection. This works by simply adding the "nameserver 127.0.0.1" directive to the /etc/resolvconf/resolv.conf.d/head file. Simple as that. Every time /etc/resolv.conf gets generated the contents of the head file are actually used as /etc/resolv.conf's header.

Using this method the local resolver is used for every connection. But Anand wanted to use only the local resolver and discard any resolvers possibly obtained via DHCP for example. Guess what, this is also possible using resolvconf.

Adding TRUNCATE_NAMESERVER_LIST_AFTER_127="yes" to /etc/default/resolvconf does exactly that. Now every nameserver directive after the 127.0.0.1 one is ignored and will not make it into /etc/resolv.conf. You can of course add more nameservers to the head file above the 127.0.0.1 directive.

Problem fixed I guess.
Don't forget to re-connect to the network or manually force re-creation of /etc/resolv.conf so the changes you made get populated. I really hope this is of use to some of you facing similar problems.

ISC dhcpd and IP assignments from a pool to specific hosts only

2011-01-01T23:01:00.001+01:00

Assigning an IP address statically to a host with a given MAC address using ISC dhcpd is quite trivial, one host entry, a hardware ethernet entry and a fixed-address entry and you are up and running.
But what if you want to assign IP addresses from a pool to only a few hosts with specific MAC addresses?

Before you ask yourself why someone might want to do that, have a look at my (very real) use-case.
I am currently working on setting up an installation server for my employer, ANEXIA Internetdienstleistungs GmbH. The server itself uses PXE, TFTP and FAI for installing systems. To be able to do PXE booting one has to set up an DHCP server to provide configuration details, like the TFTP Server Address and the boot filename.

Now what one should consider is that this system is designed to provide automatic installations for internet-facing hosts, namely ones in public IP networks. Running a DHCP server in such a network is not a good idea. We neither want to dish out configurations to each and every hosts that asks for them, neither do not want to do a PXE boot each and every time one of our systems is restarted. Now the combination of FAI and pxelinux allows for default configurations which force local booting, but this still causes the (re-)boot time for those systems to increase and potentially also increases the load on the TFTP server. Also, let's not even consider thinking about whether this setup is "clean" or not. I personally believe that dishing out IP addresses in a public IP network is a bad thing(tm) and I guess a lot of people will be nodding when reading these lines.

What I was asking myself is how to get something like that set up in a cleaner way, and guess what, I found a solution.
The basic idea behind this is only providing IP configuration via DHCP to a specific set of hosts (with a specific set of MAC addresses) and not providing any information to all other hosts. The specific set of hosts are those that we want to do an install run on. This is a no-brainer and I guess the right way to do that, but implementing this approach is not as straight-forward as I initially thought.

Actually the implementation of that idea caused me a bit of a headache and cost me a few work-hours to get right, that's why I'd like to share the configuration details with you.

Let's have a look at how to get such a setup using ISC dhcpd. We are using the fact that ISC dhcpd allows you to not only configure a subnet, but rather also pools inside subnets, which can have allow and deny rules. Such rules can be in the form of "allow/deny member of ", where classes (and subclasses, keep on reading for details) can be defined inside the configuration file as well.

What we first did was creating a subnet with a pool declaration, as follows:

subnet 10.0.0.0 netmask 255.255.255.0 {

option routers 10.0.0.254;

   option broadcast-address 10.0.0.255;
   filename "fai/pxelinux.0";
   next-server 10.0.0.254;
   server-name "10.0.0.254";
   pool {
   allow members of "install";
   range 10.0.0.10 10.0.0.230;
   }
}

This one configures the subnet 10.0.0.0/24, with 10.0.0.254 being the network gateway, 10.0.0.254 being the TFTP server and "fai/pxelinux.0" being the TFTP filename. Additionally pool allows us to define a range of IP addresses we want to use, along with a line stating that only members of the "install" class should get a network configuration. If you do not have any other subnet defined in your config and a client that is not in this "install" class asks for an IP address you will see something like this in your syslog: "dhcpd: DHCPDISCOVER from 11:22:33:44:55:66 via eth1: network 10.0.0/24: no free leases". dhcpd will not even answer these requests and thus the client will not even know that there is a DHCP server running here. Exactly what we wanted.

I wrote about this giving me a headache, but so far things have been pretty straight-forward. Getting this far did not take very long, believe me.

Next thing we did was defining that "install" class as follows:

class "install" { match hardware; }

Again, not very hard to do. This tells dhcpd to look for subclasses of "install" with a matching hardware-address. So let's have a look at the subclass for, let's say the host with MAC address "11:22:33:44:55:66":

subclass "install" 1:11:22:33:44:55:66;

I intentionally highlighted the leading "1:" there. This means nothing more or less than "ethernet". Without that leading "1:" you won't get anywhere. Matching will fail, simple as that. It took me a while to find information about this in "man 5 dhcp-eval". Quoting parts of the interesting section:

The hardware operator returns a data string whose first element is
the type of network interface indicated in packet being considered,
and whose subsequent elements are client’s link-layer address. [...] Hardware types include ethernet (1), token-ring (6), and fddi (8).

Now, with the combination of the subnet, pool, class and subclass directives we could get the setup we wanted: a DHCP server only providing IP configuration to a specific set of hosts and ignoring all other DHCP requests.

If you have any comments about this setup or ideas on how to get something similar set-up using another approach feel free to leave a comment.

Personal final note: accidentally typing 80 instead of 08 in a MAC address will cost you an additional two hours and will even have you re-compile ISC dhcpd with eval debugging turned on, believe me. :-)

What's all the fuzz about canonical-census?

2010-08-10T14:07:00.000+02:00

I know I have not updated this blog in quite a long time now, but something caught my attention today: canonical-census.

As slashdot.org reports Canonical begins with tracking their (OEM) installations. Now it's obvious that people are uncomfortable with a program running on their system which phones back to their OS vendor, that's why I have had a quick look at what exactly canonical-census does.

Firstly however, I would like to point out that the report on slashdot.org is very clear about which information is being gathered, being "the number of times this system previously sent to Canonical [...], the Ubuntu distributor channel, the product name as acquired by the system's DMI information, and which Ubuntu release is being used". And it's perfectly correct. After getting the canonical-census Debian source package (using dget -u https://launchpad.net/ubuntu/+archive/partner/+files/canonical-census_0.1.dsc) the source package shows, besides the Debian packaging information, two scripts:

census (written in Python) and
send-census (a GNU bash script).

Now what do those scripts actually do?

send-census is installed in /etc/cron.daily, which means it will be executed once a day by the system's cron daemon. It's a mere 48 lines long, and its code is quite simple. So everyone with at least some shell scripting experience can easily check what it's doing. Now guess what, it sends exactly the information as reported on slashdot to Canonical. Nothing more and nothing less.

Technically it keeps a plain text file containing a single number as its call-counter, residing in /var/lib/send-install-count/counter and uses an on my Ubuntu Lucid system nonexistent /var/lib/ubuntu_dist_channel file for getting information about the distribution channel.

The above mentioned "system's DMI information" is not the whole bunch of DMI information available, but only the contents of /sys/class/dmi/id/product_name, which strangely enough returns "System Product Name" on my machine. Last but not least it uses lsb-release to get the distribution release (ie. 10.04 for my system).

Now those four pieces of information are sent to http://census.canonical.com/submit via a simple HTTP GET query, using wget. The full URL with all the parameters added is:
http://census.canonical.com/submit?count=count&dcd=dist_channel&product=dmi_product_name&release=ubuntu_release_version

The second script, census, is the part working on Canonical's script. Basically census reads in their Apache's access log file and creates an SQLite database from the contents of the log file. With 391 lines this script is a bit longer, but it does not end up in the Debian package at all.

Personally I do not see how Canonical or one of their partners could possibly do anything harmful with that information. Comparing this to Debian's popcon reveals that Debian is gathering a lot more information.

Now there are two more things one should consider: census is targeted at OEMs, which means its unlikely that it will end up on each and every Ubuntu installation and can be uninstalled by removing the canonical-census package with your favorite package manager.

Finally, think about this for a second: It's a shell script you can always examine. There is no hidden magic and it's a plain HTTP request the script is sending. No evil things happening there.
And now compare that to what other (often proprietary) software vendors do and how much data they submit, possibly even in encrypted form so you do not know for sure what is being sent to them.

Personally I welcome the openness of Canonical with providing their users with the package's code this early and being straight about what information it submits. They could have silently added it to those installations after all...

Happy hacking!

Rest in peace Flo: 13.11.1986–16.12.2009

2009-12-20T00:15:00.000+01:00

Today is a sad day. Everything feels like I am having a bad nightmare. That's because today I learnt from the too early death of my friend Florian Hufksy.

I am sitting here and do not really know what to write. I keep on thinking about the great times we spent together. The time we started programming when we were twelve. The time we spent learning BASIC. All the times you knew more than me and could teach me a thing or two. I remember our geek talks. How we would discuss latest games. How we lost contact and how we met again. I am thinking about how sorry I am for not having met you often enough. I keep on trying to understand what drove you that far. How you could just end it all. More and more memories come to my mind, like the moment when you showed me one of your projects, Super Mario War. The moments we had playing video games together. All those moments, all that time, I miss you my friend. You were a genius, always a step ahead, not only of me, but seemingly the whole world. I can't stop thinking about your brilliant ideas and how you always finished your projects. You were a real hacker, a real genius, a person trying to make the world a better place, a person who will be missed, not only by me.

You were a genius and I always respected you, not only as a hacker, but as a beloved friend. Why did we not spend more time together? Why did you have to go? Why do I have to write this now, sitting here in my chair with tears in my eyes? And all those memories come up again and again. There is so much more that comes to my mind, but I can't keep on writing, it just hurts too much.

The world is a sad place today. I am sad. I am mourning the too early death of my beloved friend, Florian. You will always have a special place in my heart.

ujail: use cases, FAQs, part 1 & proof of concept, part 2

2009-12-09T11:16:00.000+01:00

As I ran out of time whilst writing the "introducing ujail" post on monday I would like to further elaborate on the idea, giving you some examples of possible use cases and then having a look at FAQs regarding ujail. Additionally I have created a second proof of concept that should be a lot faster, see below for more details.

Use cases of ujail

Monday's post was rather technical, so let's have a look at possible use cases today.

The main reason for both having the idea of ujail and starting working on it is my web server. I am running quite a few (S)CGI scripts there and, even though running them as different users, on a per-vhost basis, I have the impression of the whole thing being a bit insecure.

Okay, PHP does provide its famous open_basedir feature, but I am also running some Python applications which I simply cannot restrict easily. My first ideas involved adding something similar to open_basedir to Python, followed by the idea of replacing some C library functions, like fopen and friends on startup time.

Whilst the adding open_basedir to Python would have involved changing a lot of Python's internals I soon discarded the library patching idea as those could be worked around by injected code directly invoking syscalls. It didn't take long for me to notice that I have to dig deeper. The idea of ujail was born and after coming up with the proof of concept this seems to be a viable solution.

Now ujail is not only about protecting a web server from its web applications, but could do a lot more, for example:

Creating a sandbox for untrusted code (socket&file i/o emulation)
Implementing some sort of personal firewall (socket-call only emulation)
Testing applications that perform low-level system operations (read: package managers and friends, filesystem emulation)

I am sure you can come up with even more use-cases. What should be noted is that emulating a system call does not mean that one necessarily needs to emulate the whole filesystem. What can be done, for example, is patching through access to common files (libraries, executables, etc.) whilst maintaining a virtual filesystem for data that will eventually be modified. A copy-on-write approach is possible too, for example. There are multiple methods with which the multiple filesystem could be implemented, the most common would probably be using a state directory.

FAQs

There have been some questions about ujail in comments to my first post which I would like to answer. Also, I have been thinking about things that are different about ujail compared to other virtualization techniques. Feel free to add additional questions either in a comment or drop me an email: debian at sp dot or dot at.

Could you change the license of ujail to ... ?

Not likely to happen. The proof of concept's license is GPLv3 and the actual code's license will be too. However, ujail is a userspace application that does not need any modifications to the kernel so there should be no problems with porting ujail from GNU/Linux to any other system.
Does ujail work on operating systems other than GNU/Linux?

Not yet. If it's technically possible to implement the technique on other operating systems I would be happy to accept patches.
Do I need to patch my kernel for ujail to work?

No, ujail is running in userspace. The only thing it needs is Linux with support for PTRACE_SYSEMU.
How is this approach different from using LD_PRELOAD?

With LD_PRELOAD one can replace library functions, but malicious code could still directly invoke syscalls, working around this protection completely. Also, statically linked binaries cannot be restricted with LD_PRELOAD.
How is this approach different from user-mode-linux?

User-mode-linux (UML) works by emulating a full kernel in userspace and allows you to virtualize a whole Linux instance (including a new init process, etc). ujail is about providing a way of restricting a single process (and its childs) inside a running system in terms of access to syscalls and the partial emulation of those.
How is this approach different from linux-vserver?

Linux-vserver is a kernel patch and runs in kernel space, as opposed to ujail, which works in userspace.
Also, linux-vserver works similarly to user-mode-linux, providing a fully virtualized Linux instance.
Does the account running ujail need any special privileges?

No, the only restrictions that apply are those of ptrace.
Where is the code?

Right now ujail is in a planning phase, and only the proof of concept code has been written and published. The actual ujail code is yet to be written and the code will be hosted on launchpad.net.

Proof of concept, part 2

An anonymous person (who were you stranger?) added a comment to my first post, suggesting "Also, why patch the process rather than just modifying its state and trapping into the kernel?". I have had a look at this approach earlier, but it didn't work out. However, I decided to give it yet another try and created a second proof of concept. That code does not require patching any code, but only modifies the instruction pointer (eip) and the first register (eax). This should be a lot faster than patching the code.

Technically the new main loop works by calling PTRACE_SYSEMU and waiting for a notification. It then saves the instruction pointer and switches to PTRACE_SYSCALL. As before it waits for the emulated syscall to exit and at this point sets eax from orig_eax and decreases the value of the instruction pointer by the size of the "int $0x80" instruction. Another call to PTRACE_SYSCALL resumes the process. The next event is the process actually entering the real syscall and yet another one leaving the syscall again. These are resumed by PTRACE_SYSCALL and PTRACE_SYSEMU respectively. So, comparing this with the first approach we are only modifying two registers now, instead of writing to the TEXT area of the running process.

Thanks should go to the anonymous commenter for making me give this approach another try.

Questions? Criticism? More ideas? Want to contribute?

Coming to an end I would yet again like to let you know that I am open for questions, criticism, more ideas and contributions in general. So if you are interested in this topic come join the discussion by either dropping me an email, writing a comment to this post or replying to this post on your own blog.

Introducing ujail & proof of concept

2009-12-07T17:49:00.001+01:00

Lately I have been thinking about methods to provide a stripped down, secured environment for running untrusted code on GNU/Linux. With this post I would like to present you with the first results of my research.

ujail - brief introduction

I have chosen ujail as the name for the technique I am proposing. ujail stands for micro jail in userspace and, in itself, describes the concept briefly. The main idea is to have a userspace process monitor system calls of one of its childs and emulate some calls, if needed. This is done using ptrace and namely both PTRACE_SYSEMU and PTRACE_SYSCALL.
The ujail process should not be able to monitor syscalls, like strace does, but also intercept and emulate them.

This sounds a lot like user mode linux (uml), but the method is different. Whilst uml comes with a complete kernel, emulates all system calls and this way provides a virtualized system, ujail is intended to only emulate some systemcalls, without emulating the kernel.

Revisiting PTRACE_SYSCALL & PTRACE_SYSEMU

To better explain how the ujail technique works I would like to have a quick look at PTRACE_SYSCALL and PTRACE_SYSEMU again.

PTRACE_SYSCALL allows a userspace process to be notified whenever a traced process enters or leaves a system call. This means that two notifications are normally sent: one before system call entry and one afterwards. Even though one is able to change the parameters of system calls this method does not allow system calls to be fully emulated (think virtual filesystem here).

PTRACE_SYSEMU on the other hand provides one notification on syscall entry and expects the receiver of the notification to emulate the syscall. This method alone sounds great, but this also means that memory allocation needs to be emulated too, which is quite complex in userspace.

A hybrid of PTRACE_SYSCALL & PTRACE_SYSEMU

Now on to the concept behind ujail. The method I am describing works by calling PTRACE_SYSEMU for a specific process and this way taking over emulation of all system calls. However, some system calls are complex to emulate in userspace, and so a hybrid of both PTRACE_SYSEMU and PTRACE_SYSCALL is needed. In short this works by checking whether the syscall needs to be emulated when the PTRACE_SYSEMU event is received.
Now one way is emulating the syscall, filling the processes' registers and resuming execution of the process. This is simple and straight-forward.

The second way is forwarding the system call to the kernel. The problem here is that calling the syscall in the monitoring process will make the new resources available to that very process, and not the process to be jailed. This is where the hybrid method kicks in.

The proof of concept code creates a backup of the next instruction to be executed along with a copy of the instruction pointer at this point and patches it with the opcodes for "int $0x80", causing the syscall to be made again. After that it resumes execution with PTRACE_SYSCALL and waits again. The first event to be received now is the program leaving the emulated system call, which can be ignored. Resuming yet again will give use two PTRACE_SYSCALL events, one for syscall entry and one for syscall exit.

The first event is not really interesting, but at the second event the opcode backup is restored and the eip set from the saved value. Now the kernel has handled the syscall and the result is ready for the child process. A final call of PTRACE_SYSEMU resumes execution of the child and waits for the next syscall.

Proof of concept

The proof of concept code can be downloaded from its bazaar branch at launchpad.net. It is intended to be used on i386 systems only and works with simple programs, but is known not to work with anything using fork, vfork and most likely will not work for binaries using threading.

Finally, I would like to thank Pradeep Padala for his "Playing with ptrace" articles [0][1], which were fun to read and worked as a great introduction of ptrace for me.

Now there is only one thing left to say: if you are interested in this method, see loopholes or problems or want to contribute, please go ahead and contact me:

debian at sp dot or dot at

How to copy partitions under GNU/Linux the easy way

2009-12-01T18:22:00.001+01:00

After getting a new disk for my Popcorn Hour A-110 device I had to copy all partitions from the old disk onto the new one so I do not have to reinstall some applications and reconfigure everything.

After searching the web and trying to find a free alternative to Norton Ghost and Acronis True Image, preferably not using a boot disk on its own (I did not want to backup my workstation after all, just a simple partition to partition copy between two SATA disks) I gave up and decided to do the copying manually.

So I fired up gparted to do the partitioning, did a right click and... I noticed that gparted supports copy/paste. Being curious about what this could potentially do I gave it a try. I marked partition one on the old disk, did a copy, went to the new disk and clicked on paste - and guess what, gparted did what I was looking for.

Putting a long story short: you can copy whole partitions using gparted's copy/paste mechanism and even resize them whilst doing so. I am somehow ashamed I did not notice this feature earlier, having been a gparted user for a few years now and I can imagine I am not the only one who missed that.

kvm, qemu and the magic of ubuntu-vm-builder

2009-11-11T09:25:00.001+01:00

As I noted two days ago I was unable to build Android on Ubuntu 9.10 x86-64 and thus needed to set up a virtual machine.

At first I went for my preferred virtualization solution, VirtualBox and had to notice that even though I assigned all 4 processor cores of my workstation (along with 2GiB of memory) to the virtual machine building was painfully slow. I immediately ditched the idea of using VirtualBox again and decided to give something new to me a try: the combination of kvm and qemu.

Having Intel VT-x support built into my workstation's processor I thought that this combination should give better performance, and I wasn't disappointed. To be honest, I am astonished on how fast the beast is now. Disk speed still seems to be not as fast as running things natively, but there must be a downside somewhere. :-)

After a bit of googling I also found that ubuntu-vm-builder exists, which simplifies virtual system creation tremendously.

My Android working tree is being synchronized right now, which means that I should be able to start building in a few minutes time. I hope the virtual machine stays as fast as it is right now during the build and I hope everything goes well.

An update on the proprietary Maemo SDK installer

2009-11-09T22:00:00.003+01:00

Yesterday I wrote about my dissatisfaction with the current state of the Android 2.0 code tree and how a proprietary install script for Maemo scared me off.

As suggested in one of the comments to my post I filed a bug report against Maemo, bug 6087.

Besides getting quite a few replies to the bug report within a matter of hours Carsten Munk pointed me at Maemo SDK+, which has less restrictive licensing.

Another comment, by Marius Gedminas (thanks!) pointed me at Mer,

a new operating system for small, mobile touch-screen devices.
It is Linux based and layers the best open-source elements of Nokia's Maemo platform over a modern Ubuntu base.
The goals of Mer include:

Integrate the best solutions for a wide variety of small form-factor devices
Encourage wider access to device capabilities through the Vendor Social Contract
Demonstrably provide an easy route to market for vendors
Dramatically reduce costs to vendors of supporting EOL hardware
Focus, harness and support community contributions to the platform
Encourage and ease migration of existing applications
Support experimentation, innovation and development

My Android repositories

2009-11-09T21:58:00.000+01:00

As I wrote in my last post I noticed a few problems with Android's roaming detection code and decided to try fixing it myself.

So, I am basing my work on CyanogenMod, which I am also using on my Android device. My repositories are hosted at github.com/speijnik and you can fetch (nearly) everything you need for building by using repo. See the README file in my android repository over at github for details.

For now only the simplification of the roaming detection code has made it into the repository, but be aware that even though I have published the code I still have neither built nor tried it, as I do not have a working build environment set up yet.

Oh, about the working build environment: there seem to be problems with either the webkit code in the Android repositories (unlikely) or with building that code on Ubuntu 9.10 x86-64 (more likely). Right now I am downloading Ubuntu 8.04 LTS i386 for use in a virtual machine. I will let you know whether that fixes my problems or not.

Android's roaming detection & its implementation

2009-11-08T21:04:00.000+01:00

I know I wrote about Android already today, but there is another thing that concerns me right now. I am owner of an Android-based phone (an HTC Dream) and recently switched my mobile network provider. The problem is that my new provider is a virtual provider and as such there is no real network of that provider. Now Android has a feature to turn off broadband connections when in roaming mode, which itself is a great idea and can save you from paying quite a lot of money when the phone connects to 3G abroad, but this feature also turns off broadband connections when roaming locally. All this is being discussed in bug report #3499.

After noticing this problem I became curious on how Android detects that it is roaming and I found the GsmServiceStateTracker.isRoamingBetweenOperators method to be responsible for that magic, but soon noticed that the method is not only inefficient, but also doesn't work as intended. This is hardly related to the bug mentioned above, but let's have a look at the code in question:

/**
* Set roaming state when gsmRoaming is true and, if operator mcc is the
* same as sim mcc, ons is different from spn
* @param gsmRoaming TS 27.007 7.2 CREG registered roaming
* @param s ServiceState hold current ons
* @return true for roaming state set
*/
    private
    boolean isRoamingBetweenOperators(boolean gsmRoaming, ServiceState s) {
        String spn = SystemProperties.get(PROPERTY_ICC_OPERATOR_ALPHA, "empty");

        String onsl = s.getOperatorAlphaLong();
        String onss = s.getOperatorAlphaShort();

        boolean equalsOnsl = onsl != null && spn.equals(onsl);
        boolean equalsOnss = onss != null && spn.equals(onss);

        String simNumeric = SystemProperties.get(PROPERTY_ICC_OPERATOR_NUMERIC, "");
        String operatorNumeric = s.getOperatorNumeric();

        boolean equalsMcc = true;
        try {
            equalsMcc = simNumeric.substring(0, 3).
                    equals(operatorNumeric.substring(0, 3));
        } catch (Exception e){
        }

        return gsmRoaming && !(equalsMcc && (equalsOnsl || equalsOnss));
    }

Okay, let me summarize what this piece of code does wrong, at least from my understanding:

It takes both the network operator alphanumeric identifier and alphanumeric long identifier and compares both to the alphanumeric identifier coming from the SIM card, whilst...

... it could simply use the network and SIM card numeric identifiers and compare those, which should be a lot cheaper than comparing those strings

Then it takes the first three characters/digits of the numeric identifiers (which indicate the country) and compares those

Now in my case my SIM card doesn't seem to provide the phone with a alphanumeric identifier, so the first two comparisons always fail for obvious reasons and, looking at the inline-if in the last line of that method my phone will always indicate that I am in roaming mode, even when I am not.

The problem is not only the logic which seems to be wrong, but I rather see the inefficient comparisons used there to be a major problem in embedded systems like mobile phones. This is the first piece of Android code I have had a look at, but if all other code is as ugly and inefficient as these few lines Android really needs some major fixes. Related to this I have reported bug #4590 and forked the git repository in question over at github, to fix this method, should be a matter of 5 minutes.

Android, Mythbusters and openness

2009-11-08T05:49:00.000+01:00

I have been reading a great many posts about Android lately, some consisting of criticism, some of praise and some simply addressing issues in the Android "community". Let's have a look at those.

Matt Porter's Android Mythbusters presentation and Harald Welte's reaction

I haven't seen the presentation live, but I had a look at the slides. Impressing work done by Matt putting all this information together. However, we all knew that Android only (ab-)uses Linux, without making use of the GNU userland for a long time, didn't we?

In his presentation Matt has shown things such as Android's udev "replacement" that uses hardcoded values for device node creation and (on his blog) Harald has then come up with a statement I have found to be very strong:

The presentation shows how Google has simply thrown 5-10 years of Linux userspace evolution into the trashcan and re-implemented it partially for no reason. Things like hard-coded device lists/permissions in object code rather than config files, the lack of support for hot-plugging devices (udev), the lack of kernel headers. A libc that throws away System V IPC that every unix/Linux software developer takes for granted. The lack of complete POSIX threads. I could continue this list, but hey, you should read those slides. now!

Now both of these statements target technical details, but the root of the problem seems to be elsewhere.

Where is my Android 2.0?

Okay, that heading might not be making any sense in the context of this post at a first glance, but let me elaborate on that. Google and the Open Handset Alliance refer to Android as being an "Open Source" operating system, but the project is different from "real" Free Software projects: development takes place in a closed group and the results are shared with the community later on, when they are deemed to be ready.

This means that innovation also takes place behind closed curtains and that the community is not involved in the actual development process at all. Lately we have seen the result of that, as Motorola is bragging about working close with Google on Android 2.0 ("Eclair"), but the AOSP source trees, open for everyone to have a look at, show no signs of version 2.0. In fact no changes that might even remotely suggest the release of a new major version have been made public in the past few weeks. So where is the openess there?
Actually, the Motorola Droid has already shipped with Eclair on 6th, but still, there is no indication that Eclair will be made available to the broader public.

In short Android seems to be developed behind closed curtains, with hardly (read no) community input whatsoever and is sometimes released as Free Software, not what I would describe as an open development process.

The Android Market problem

As we have seen in the past Google is enforcing their copyright on proprietary applications that ship with pretty much every Android device, such as the Android Market. This has become really clear when Steve Kondik received a cease and desist letter when packing the Google-proprietary applications into his ROMs. Okay, it's Google's right to enforce their copyright and there is nothing wrong with actually doing so, the thing I really have a problem with is something else: the Market is proprietary.

Now what this means should become rather clear. You can have an Android device without Google's proprietary bits, but with default settings you just do not have any way of installing additional software. In my opinion the Market should be freed by Google themselves, or the community has to react and come up with a free replacement to overcome the vendor lock-in. Oh, you might know a replacement called SlideMe (or Mobentoo) already. Well, that bugger is proprietary too, so not a solution at all.

Nokia and Maemo to the rescue

In most discussions about the openness of Android someone throws in Nokia and Maemo, as a solution to the dilemma. Reading all those positive comments I simply had to give it a try, but all my hopes were destroyed within a few minutes.

Let's start with the good news and let alone the reason why my hopes were destroyed for another minute or two. Maemo is based on Debian GNU/Linux and various Free Software components, such as GTK+, gstreamer, esd and friends. Most of the system is Free Software which is a good thing(tm) and reading all of this really got me into Maemo. Okay, some applications seem to be proprietary, but I am sure that could be fixed rather easily, so I could once for all use a truly open phone.

...and then came the SDK installer shell script:

#!/bin/sh
# Copyright (C) 2006-2009 Nokia Corporation
#
# This is proprietary software owned by Nokia Corporation.
#
# Contact: Maemo Integration <integration@maemo.org>
# Version: $Revision: 1110 $

Now there is one question you should ask yourself: Why would someone trying to promote his platform as being open make the *installer* script for its SDK proprietary? Come on, it's an installer script, how much of your secret juice could be in there? What's the problem with people modifying it and working on this installer script in an open development environment?

I had high hopes for Nokia actually doing a bit better than Google, but it seems they've failed to do so. It may be me overreacting, but a proprietary SDK installer shell script scares me enough not to install the SDK and have a look at it for now nor to think about buying a Maemo-based device in the near future. Please Nokia, either get the facts straight or provide us with a free SDK to your free & open platform.

So, in short, Google is bad at working with the community and creating a truly open development process, and Nokia simply fails in terms of not scaring off prospective developers for their open platform with the proprietary SDK installer. Do you have any solutions in terms of an open phone environment, apart from what OpenMoko has come up with?

How to move panels in Gnome 2.28

2009-11-04T11:04:00.000+01:00

I just installed Ubuntu Karmic Koala on my workstation and came across the problem of not being able to move/drag Gnome panels around in order to have the panels on my primary monitor.
On the Debian system that was powering the workstation before this was a non-issue as I could simply click, hold and drag both the upper and the lower panel, but this didn't work.

So, after a few minutes of googling I came across an entry at answers.launchpad.net[0] and a blog post, but I cannot seem to remember the URL to that one. I can imagine that some of you might be having the exact same problem, so the solution is holding down the ALT, whilst dragging as usual.

[0] https://answers.launchpad.net/ubuntu/+source/gnome-panel/+question/264

Automagic bug reporting in Python applications for Debian

2009-08-17T21:23:00.000+02:00

We all know this situation: a program crashes and you need to send a bug report to the DBTS. The damn bug however is hard to reproduce and you fail to do so and hence can't submit the report.

This has all changed for update-manager now. With the next upload to unstable update-manager will get automagic bug reporting. In short: there is code that detects uncaught exceptions, asks the user if he or she wants to file a bug report and then invokes reportbug. Nothing too special about this yet. There is one thing that should make lives of both bug reporters and developers easier though: the code automatically includes traceback information, that make finding the cause of the problem a lot easier.

Okay, enough of praising this feature of update-manager, this post is about something else. Ubuntu users and developers might think "apport" now, because apport is an application that provides exactly this, reporting of bugs on program crashes, for all users.

At least for Python applications and libraries in Debian providing this functionality should be easy. The only thing one has to do is create a sys.excepthook implementation that does the bug reporting, just as in update-manager.

The questions I have now are:

Do you think this feature would be a good addition to the Debian distribution?

update-manager weekly update #6

2009-07-09T20:07:00.000+02:00

So finally I have the time to provide you with a weekly update, instead of my usual bi-weekly ones.

Unfortunately I did not work on anything on last week's TODO list, but found other issues I worked on and corrected. So let's have a look at what I've done.

Debian packaging update

I have done some work on the Debian packaging, which allows update-manager to be built using dpkg-buildpackage now. The way packages are splitted is not finalized yet and not up-to-date with my (and my mentor's) idea of how we should do that. You can expect an update to that soonish.

Automatically invoking package list reloading / update check

There is a command line switch (namely -c, or --check) now, that automatically performs an update check on startup. This gives other programs, like software-properties, a way of forcing a check when, for example, the package list sources have changed.

Checking/unchecking all updates in Gtk frontend

Finally the small feature of selecting or deselecting all updates works in the Gtk frontend. Special cases like "all updates already checked" or "no updates checked" yet are handled too, meaning that you can only use one of these methods if it actually makes sense.

Package dependencies in python-apt backend and Gtk frontend

Both the python-apt backend and the Gtk frontend are now aware of package dependencies. This means that when you select an upgrade that depends on another one that other update is selected too. The same works vice-versa too. Additionally the UI now lists all dependencies and dependencies on packages that are not installed yet and automatically deselects all updates that would requires new packages to be installed.

Displaying of overall download size in Gtk frontend

There has been a missing feature (ok, maybe a bug) so that the displayed download size would not be updated in the Gtk frontend. This has been fixed.

Install button being set sensitive correctly in Gtk frontend

In the past the install button would be set to either sensitive or insensitive at startup and not updated afterwards. That means if there were no packages to update when starting update-manager, then checking for updates where new updates are found, the install button would not be set sensitive again. I fixed that too.

Sorting of packages in Gtk frontend

In the Gtk frontend packages were not sorted at all, which meant that finding a specific package was rather hard. I added code that sorts the update list by package name now, which solves this issue.

Bugfixing humanize_size

The humanize_size method, which is responsible for human-readable size displaying in the Gtk frontend contained a major bug so that sizes were rounded. Again, I was able to solve this.

Next week's TODO list

As I didn't find time to work on last week's TODO list my new TODO list is in fact my old one, with additional "Bugfixing" and "Debian packaging" tasks:

Downloading and installing of updates

Bugfixing (?)

Debian packaging

Checking that everything is documented

Even more unit tests

Pylint checking

If time permits and everything else works correctly: working on an aptdaemon backend

The next thing you can expect me to update is the Debian packaging and the documentation, which are my highest priority tasks for now, followed by support for downloading and installing updates.

Happy hacking!

update-manager weekly update #5

2009-07-02T12:29:00.000+02:00

Firstly I have to apologize again for not providing you with weekly update #4, but again I didn't have the time to write one, so this post is going to sum up everything that happened since my last update.

Let's have a look at my previous TODO list:

Documentation

Even though my TODO list entry contained a more detailed entry I have updated the UpdateManager documentation as a whole, leaving only a few blank spots right now.

Ubuntu distribution specific code

I implemented changelog fetching for Ubuntu, which works just as fine as its Debian counterpart now.

More unit tests

There are plenty of unit tests now, but not everything is being tested yet. I am especially proud of my Python interface validation code, that is being used in unit tests to check if handlers implement an interface correctly.

Update list downloading

Checking for updates is what caused me major trouble in the past few days. Basically I had all the code ready, but for some reason the UI froze, with no apparent reason.
However, today I was able to finally identify and fix the problem. As I expected my code was just fine, but python-apt was messing up. I am going to discuss the exact problem and its solution later on, but first: a screenshot. :-)

Note: As you probably noticed I replaced the default progressbar with a pulsating one, because we cannot get exact information on how many items/bytes to fetch and would likely get a progress bar moving backwards, which isn't beautiful.

Further changes

The TODO list was rather short and I did a lot of other work, which I want to elaborate on.

Dynamic selection of frontend, backend and distribution specific modules

Even though this is probably not of any interest to John Doe, it helps a great deal when debugging code as all three components can be selected via separate command line switches now.
Additionally some magic has been put in place that automatically detects the system's distribution and loads the corresponding distribution specific module. This is done via lsb_release and the newly introduced code in UpdateManager.Util.lsb.

Pylint cleanup

Just out of curiosity I decided to start a pylint run on the codebase and quite a few problems were detected, which I then fixed. To be honest though I added quite some code afterwards that probably needs pylint checking and fixes again.

update-manager IPC

My original plan and IPC design involved using callback functions and passing them between the different modules. Even though this worked out fine I had the feeling this wasn't clean enough and decided to ditch this approach and replace it with handler classes.
The handler base classes now provide an interface of methods that are called on certain events and their implementations act accordingly. The main benefit was that I could easily drop a lot of enums and rather have different methods handling different events.

Gtk, threads and python-apt

With the new IPC approach it became easier to use threads that do the actual work in the background, which I had implemented in next to no time, but a few problems showed up.
Whilst cache reloading from within a thread worked just fine checking for updates did not, and until today I didn't know why. I spent a good amount of time debugging this issue, even using python profiling, but nothing obvious showed up. The background process was running, whilst the UI froze.
Today I finally found the root of the problem: python-apt. Even though I assumed that the python-apt worker threads must be stealing CPU time from the thread running gtk.main I wasn't sure how this could be happening, having two completely independent threads.

Now, the cause of all this mess was that Python has a global threading lock and it seems as if this one is *LOCKED* when running C-code, such as the one python-apt comes with. The solution lies in calling Py_BEGIN_THREADS_ALLOW and Py_END_THREADS_ALLOW from within the C code, to release the global lock and let the Python interpreter do some work every now and then.

As with the python-apt acquire code I was able to allow other threads to work as soon as the fetching code starts working and only disallow threads when actually modifying Python objects or calling methods and/or functions. Surprisingly python-apt already made use of this in its cache loading code, but not the fetch progress code.
Fixing this problem took me less than half an hour and you probably can't believe how glad I was to finally get things working again.

UI updates & other changes

Some details in the UI were anything but optimal, like horizontal scrollbars in a few places, which I removed. Additionally I saw the need to move some code out of the Gtk frontend's __init__.py file and to a separate ui.py file.
A full list of all changes I made is available from the bzr changelog at bzr.debian.org.

A few more screenshots

Finally, I would like to provide you with two more screenshots (don't worry about my system being insecure because of not applied updates - this is a testing machine that is not up-to-date on purpose):

TODO list

My TODO list for next week:

Downloading and installing of updates

Checking that everything is documented

Even more unit tests

Pylint checking

If time permits and everything else works correctly: working on an aptdaemon backend

Python interface validation

2009-06-22T23:30:00.000+02:00

When I started working on update-manager I thought using zope.interface for my interfaces was a good idea, but soon realized that it lacked a way of actually validating a given interface against an implementation. The only thing it did was checking whether the implementation defined that it implements the interface.

Now, whilst writing some unit tests for update-manager I came up with a simple way of doing "real" validation, and I would like to share that Python code with you.

Firstly, I'd like to give you an overview of which checks my code carries out:

Mandatory method (raises NotImplementedError in interface definition) is not implemented (also raises NotImplementedError in implementation)

Optional or mandatory method is of correct type (static method versus instance method)

Optional or mandatory method has a different signature (argument count is different)

I consider at least the first and last check viable for validation of an interface against its implementation. The second check I listed is not that useful, and may produce false positives when someone uses certain decorators, I did not carry out any tests on that myself though.

The code can be found in update-manager's repository (link) and (for now) is licensed under the GPLv2 or later. I am willing to distribute this code as a separate Python module (maybe under a more permissive license like the LGPL) if enough (let's say at least two) people are interested in it, so please let me know if you like it.

Apart from the code itself the unit tests in the file linked above should explain how this beast exactly works.

Happy hacking!

update-manager weekly update #2

2009-06-19T04:27:00.000+02:00

First of all: yes, I skipped update #1. I was rather busy with some assignments and exams at university and didn't work that much on update-manager the past two weeks.

Anyways, this update contains everything that has happened since update #0.

Changelog fetching

The changelog fetching code has been added to update-manager. This means that the changelog will be shown in the details section now and should look the same it looked before. However, I have only written that code for Debian so far, but the Ubuntu part is on my TODO list.

Documentation

The documentation has been updated and uploaded to alioth and can be viewed here. I have set up a python environment on alioth which allows building the documentation directly, rather than building it locally and uploading it then. Basically this works by having a separate python packages directory, containing some mock modules that are needed (think gtk and friends here), allowing us to build the docs without having to install all dependencies.
I am planning on elaborating on this method and how to create such an environment in one of my upcoming posts, so stay tuned if you could use something like this too.

Additionally to this environment the documentation has been updated a great deal, including more modules and containing documentation for previously undocumented methods and classes.

Application module

I have reworked some aspects of the UpdateManager.Application module, allowing me to do unit testing on pretty much every aspect of the class. The problem I fixed here is that Application directly called sys.exit when something went wrong and now raises exceptions, which contain the status code and are handled in the respective scripts (ie. "update-manager").

Gtk Frontend and updates from another thread

One thing I fixed was the problem caused by the changelog fetching code running in a separate thread and invoking a callback function that updates the UI. It seems as Gtk isn't that happy when you do this and the UI wouldn't be updated immediatly (it seemed that this only happened after some events, like scrolling the update list). This has been reworked and the callback function now checks if it was called from the main thread or not and calls gtk.gdk.threads_enter/_leave accordingly.

Changelog Viewer

After finishing the changelog fetching code I added the ChangelogViewer widget from previous update-manager versions again, supporting creation of links to launchpad and debian bugs (ie. LP:NNNNNN and Closes: #NNNNNN are now links) and displaying the version number in bold, among other things.

Weeding out UpdateManager.Frontend.Gtk.utils

Initially I just copied over the utils module from old update-manager to the new implementation, leaving every single function in there, but now I decided to weed out the module. The result is that only the functions actually used by this implementation remained in there. Related to this documentation of that module is pending and on my TODO list.

Version number

After a chat with my mentor we decided to bump update-manager's version to 0.200-pre. This should make it easier to distinguish from the old version and indicates that a lot has changed. The first release following the -pre series will be 0.200.0, which should then include all functionality old update-manager included.

My TODO list for next week

Ordered by priority

Documentation of UpdateManager.Frontend.Gtk.utils and .ChangelogViewer modules

Ubuntu Distribution Specific code

More unit tests

Update list downloading in Gtk frontend

Should CLI debug output and error messages be localized in a GUI application?

2009-06-02T08:55:00.000+02:00

Whilst working on update-manager I have been wondering whether I should use gettext for localizing debug output and error messages sent to stderr.
As for debug output itself I basically do not see the need for providing a localized version for each and every message sent to stderr, but as far as error messages are concerned I am uncertain.

The point is that update-manager (apart from its experimental text interface) is usually not launched from a terminal at all and so most users won't even see these messages ever. Also, I believe that every developer's English skills are good enough so that he or she is able to understand simple messages.
Error messages however might be useful to all users when they experience a problem with the software, but localizing those could make handling bug reports a bit harder, possibly having to translate the error message back to English before being able to see what has gone wrong.

So basically I am asking you: What do you think? Is it worth localizing these messages? What is your experience with localized or non-localized error and debug messages?

I would be glad if I could get some input from you, either as a comment to this article, via email to debian(dot)sp(dot)or(dot)at or through the update-manager-devel mailing list.

sphinx-aware Enums in Python

2009-06-01T20:20:00.000+02:00

As I promised to keep you updated on recent developments on update-manager I am writing this article. Just as a disclaimer: I am not going to write about any recent developments here, but would rather like to point at a piece of code I added to update-manager that could be useful in other applications too.

Now, as the title suggests there are sphinx-aware Enums in update-manager. Enums are common constructs in other programming languages like C and allow simple creation of constants with, for example, ascending values (first constant has value 0, second has value 1 and so on). Python unfortunately does not include support for Enums itself, but I found it rather easy to write classes that emulate such a construct.

Nothing is new about Enums in Python and there are probably quite a few different implementations out there, but I believe mine is different. The sphinx-aware part means that my implementation automagically updates the docstrings of the created instances and thus allows sphinx' "autodata" method to include sensible information in generated API documentation.

I could go on writing about and praising my method, but I believe a short example gives you a better idea how my implementation works and what I wanted to achieve with this. Have a look at this page, which is part of update-manager's new API documentation. You should see rather well-looking documentation of the UpdateManager.Backend.RELOAD_CACHE_STATUS NegativeEnum, the defined constants, their values and some additional information about each value now.

Still, nothing too fancy, HTML documentation generated from docstrings. What makes this special is the code from which it was generated:

RELOAD_CACHE_STATUS = NegativeEnum(
  BEGIN = "Started reloading package cache",
  DONE = "Finished reloading package cache")

This not only gives us a RELOAD_CACHE_STATUS enum, along with the RELOAD_CACHE_STATUS.BEGIN and RELOAD_CACHE_STATUS.DONE, but also some documentation, included in RELOAD_CACHE_STATUS' docstring, that can be used by sphinx.

You can find the Enum code, which is rather short and should be quite easy to understand, here. I hope you find this code as useful as I do.

update-manager on alioth

2009-05-28T13:44:00.000+02:00

As I noted in this weeks update-manager progress update one of my tasks was to create an alioth.debian.org project and get my branches uploaded to Debian.

I did not imagine that alioth admins (hi there, a huge "thank you" goes to you guys) would be this fast with reviewing and accepting the project and enabling bazaar support for me.
Anyways, the project has been accepted and its new home is on alioth. I have also already uploaded both my update-manager branch and python-apt branch to bzr.debian.org

Additionally I have generated the API documentation, which is also hosted on alioth, and created a development disccusion mailing list, update-manager-devel at lists.alioth.debian.org.

If you are interested in this project feel free to have a look at what I've done so far and join the development discussion. Comments, critizism and ideas are always welcome.

update-manager weekly update #0

2009-05-28T01:52:00.000+02:00

It has been more than a month since I last wrote about my work on update-manager during this year's Google Summer Of Code and I am somewhat ashamed I wasn't able to provide you with updates more regularly.

So first of all, yes, I did do some work and yes, there has been quite some progress. Basically both private and university stuff have kept me from writing and that's why I'd like to start with this series of weekly updates today.
This series are meant to summarize what has happened during a week of writing code and give you an overview of what's happening. This first issue however will sum up the past month.

So let me begin explaining what has happened since my last post.

update-manager bazaar repository

All code I have written so far is available through a public bazaar branch on launchpad.net. My branch's page can be found here and provides you with its history and of course instructions on how to obtain the code. The location is only temporary though, as I am going to move hosting over to alioth.debian.org. This is on my task list for next week.

modular design

I have ripped apart nearly all of update-manager and put it together in a more modular way, which should implementing new frontends or backends more easy, whilst also simplifying code maintenance.

The new design consists of four major parts:

The application class is responsible for parsing command line arguments, initializing all other components correctly and coordinate communication between the frontend and the backend.

The backend itself is defined through the UpdateManager.Backend.BackendBase class and each implementation subclasses BackendBase. It is responsible for interacting with apt.

The frontend is again defined through a base class, UpdateManager.Frontend.FrontendBase. This part of update-manager provides the userinterface, handles user input and starts operations accordingly.

Lastly there is the distribution-specific part, which lives inside the UpdateManager.DistSpecific Python module and is defined by its own base class, DistBase.

backend implementation

When I started working on update-manager it heavily relied on synaptic and used it to do the dirty-work. However, together with my mentor, mvo, I decided to drop synaptic support and rather concentrate on using python-apt. This means that the only backend implementation right now is a python-apt backend.

The python-apt backend is currently a work in progress, but already includes some basic functionality. Right now it can (re-)load the package cache and package lists and is able to provide a list of packages which are upgradable to the frontend.
Whilst implementing these functions I noticed some shortcomings of python-apt itself, fixed those and got mvo included in his python-apt branch at launchpad.

frontend implementation

I started re-implementing the Gtk frontend as provided through current update-manager and right now it visualizes the package cache reloading process and provides users with a list of upgradable packages. However, that's pretty much all of the functionality it includes right now, which is why implementing more functions is pretty much on the top of my todo list.

Additionally I have ported the text frontend, as included in Ubuntu, to the new modular system, and this frontend's code really shows how easy adding a frontend with the new modular design is. This frontend contains the same functionality as the Gtk frontend.

distribution specific code

The core described above does not include any distribution specific code anymore, which is the main focus of this project. The implementations of distribution-specific functionality contains classifiers for update categories for both Debian and Ubuntu, whilst I focused on getting things right with the Debian implementation for now. These classifiers allow the frontend to let the user know which kind of update they are about to install, like a security update, a recommended upgrade or a third-party (unofficial) upgrade.

documentation

As update-manager was poorly (read: hardly at all) documented I started documenting the API using sphinx. However, right now the generated documentation cannot be found anywhere yet. This should change as soon as an alioth project for update-manager has been created.

next week's tasks

I would also like to provide you with my task list for the coming week. The list, ordered by priority, is:

Generate an HTML version of the API documentation and put it on alioth.

Implement changelog-fetching for the Debian-specific module and make use of that from within the code and the Gtk frontend.

As you can see this list is rather short. This can mainly be attributed to a few university assignments, and instead of providing a long list of tasks which I probably won't be able to finish I rather keep the list short and hopefully get things not on this list done too.

update manager to become more modular

2009-04-24T19:41:00.000+02:00

In my last post I wrote about how I got accepted for GSoC09 and am going to work on update manager. Now I couldn't wait for the actual GSoC09 coding period to start and created my own update manager branch right away and started hacking.

So far I have only written a few lines of code, but my mentor Michael Vogt and me came to the conclusion that whilst working on the internals of update manager it might be a good idea to make the whole program more modular.
Right now all the different functions of update manager (being the UI/frontend and the package manager interface/backend) are mixed up in various files, which makes not only reading the code harder, but also extending update manager more difficult. This was reason enough for me to have a look into making update manager more modular in its design and some of my efforts can already be seen in my update manager branch.

If you have any comments on the proposed backend interface or see major problems with it, please let me know, I would really appreciate some input on that. Also, the UI and the distribution-specific code interfaces are next on my list, before beginning to actually move existing code around. I hope to be able to finish that work before the GSoC hacking period starts, so I can concentrate entirely on my task of making update-manager distribution independent.

Summer Of Code 2009: Working for Debian

2009-04-21T09:10:00.000+02:00

Yesterday Google announced the students and projects that have been accepted for Google Summer Of Code 2009 and guess what: my project was accepted. This means I will be working full-time on FOSS this summer.

So I guess it's about time to introduce my project to you: Distribution-independent update manager, mentored by Michael Vogt (mvo).

Okay, I believe some of you might wonder what this project is all about, as update-manager is in the Debian package archive already. There is a problem with update-manager though. As you see in the package's version number (it contains ".debian") update-manager has been adapted for use in Debian. Also, Debian contains update-manager 0.68 right now, whilst upstream (Ubuntu in this case) has released 0.111.6 (actually there were quite a few upstream versions meanwhile). The reason Debian is nowhere near being up-to-date with upstream is that right now a lot of effort has to be put into porting update-manager to Debian every time a new upstream release is made, because certain Ubuntu-specific functionality breaks update-manager in more or less severe ways on Debian.

This leads me directly to what my project is about: making update-manager (Ubuntu-) distribution-independent, but not package manager independent.
There are 6 main goals for this project, which I will be working on in the order below.

Analyzing the code and identifying Ubuntu-specific parts.

Creating a distribution-plugin interface and moving the Ubuntu-specific parts into a distribution-plugin, creating a core package that is distribution-independent.

Creating a special notification for important/security related updates and providing the code that handles updates from security.debian.org as such.

Creating a backend-plugin interface, moving the synaptics backend into a backend-plugin and optionally create a python-apt based plugin.

UI redesign, providing a simpler interface to average joe, whilst allowing more experienced users to optionally display more information.

Automatic downloading & installation of updates. This is still up to discussion, as automatic downloading is already provided by software-properties (-gtk and -kde) and automatic installation can be handled by unattended-upgrades. Both packages are part of Debian already.

Please note that this list should not be considered final and may be extended or modified over time. It exists to give you an overview of what exactly my project is about and how I am planning on carrying out the tasks.

Finally I wanted to let you know that I will keep you posted on the progress I am making, via this blog. Alternatively a blog aggregator for Debian's GSoC students has been set up over at http://soc.alioth.debian.org/feeds/blogs/, where you can not only find my posts, but those of all of Debian's students.

Python everywhere: computer games

2009-04-02T19:35:00.000+02:00

This is the second article in my series Python everywhere and covers the use of Python for in computer games. The first article of this series covered the use of Python for the conficker worm scanner tool and can be found here.
Games written in Python

As PyGame provides a nice library for writing games purely in Python it is becoming more common to use Python for this task too. The book "Beginning Game Development with Python and Pygame" is linked directly from the PyGame homepage, and thus is probably a good resource if you want to start writing games in Python.

However, I do not want to go into detail on how this library works, but rather provide you with a few examples of games written in Python. To provide you with a few examples I had a look at the PyWeek homepage. PyWeek is a Python Game Programming Challenge which invites everyone to participate, so the winners of this contest are of high-quality, and I'm showing you the latest two winners.

There are always two winners of PyWeek in for indivduals who have created games and teams. The latest winners are "Team Rambo" in the individual effort category and "Midnight Sun" with their two-man team.

PyWeek: Team Rambo's Stringrolled (individual)

Stringrolled makes use of the pygame library I mentioned earlier and is a platform game. In a mere 2377 lines of code, including comments and blank lines, Team Rambo created an impressive game, coming with a story, easy-to-learn controls and nice 2D-graphics, screenshot below.

PyWeek: Midnight Sun's Kite Story

Kite Story is yet another interesting game, with game mechanics I have not seen ever before. You are controlling a kite with your mouse and are trying to catch objects, such as bees and birds, with the kite's rope. So what you basically do you draw a loop around an object with your mouse and that way catch it. Every third cought object you advance to the next level, but keep in mind not to collide with the objects, because you will lose them and in turn be doing the previous level again, screenshot below. It should be noted that this game does not make use of PyGame at all, but rather relies on pyglet, and is 1997 lines of code in length, again counting blank lines and comments too.

Games using Python

You have seen now that it is possible to write a game completely in Python, but there's another use-case of Python in games: scripting.
Some (proprietary) games, such as Civilization IV, offer Python support in their editors and SDKs. This quote from the article at 2kgames.com should give you a good idea of what can be done using Python in Civilization IV:

The next level offers Python and XML support, letting modders with more experience manipulate the game world and everything in it. XML (eXtensible Markup Language) files can be edited in standard text editors or in special XML file editors that have ease-of-use features like a grid view. Editing these files will allow players to tweak simple game rules and change or add content. For instance, they can add new unit or building types, change the cost of wonders, or add new civilizations. Players can also change the sounds played at certain times or edit the play list for your soundtrack. NOTE: You can have custom soundtracks simply by adding music to the custom folder. You only need to edit the XML in order to assign certain pieces to specific eras or remove certain pieces.

The Python scripting language is fully integrated throughout the game and offers experienced modders a chance to really strut their stuff! People with some programming skills will be able to do things to alter the game in interesting and extraordinary ways. For instance, all of the game interface screens are exposed to Python, so modders will be able to change the information that's displayed, as well as how it's positioned on the screen. We also use Python to create and generate all of the random map scripts that are included in the game. So, players will now have the ability to add scripted events to the game like automatically generating units when a tile is reached, having specific situations trigger automatic war, or get this, bringing back Civil Wars caused by unrest, Civ II style!

EVE Online is another game making use of Python, as an article over at eveonline.com points out.

Python everywhere - also in compuater games

Even though I am sure you can come up with a lot more examples of Python being used in computer games I think I have proven my point. Python is being used not only to create computer games, but sometimes also to provide developers with a way of extending games. To me personally it feels as if adoption of Python for this very task is increasing too, and I expect Python to be used even more by the game development community in the future.

You can expect the third part of this series to be released in about a week, so please check back regularly if you like the series.

Problems running PHP as a separate FastCGI process

2009-04-01T14:06:00.000+02:00

As some of you might have noticed this webserver has not been that responsive in the past few hours and I have been working hard on getting that fixed. I finally identified the problem and was able to fix it.

The root of the problem was my setup running PHP as a separate FastCGI process. Unfortunatly it seems as if PHP can only handle 500 requests per FastCGI process and then seems to lock up.
The old setup of this site didn't cause such problems and it seems the problem lies in not setting the PHP_FCGI_CHILDREN and PHP_FCGI_MAX_REQUESTS environment variables with the new setup.

I initially thought that the default values of those environment variables were safe, but they are not. As I already wrote PHP seems to lock up after 500 requests and the solution lies in changing PHP_FCGI_CHILDREN, which defaults to 0 (no additional processes) to something bigger than 0 (I am using 2 children to make sure I have at least one PHP process reading for answering requests at any time).
Why? Quite simple, if you increase the value the PHP root process becomes some sort of manager and delegates requests to the children, as expected. However, using PHP_FCGI_MAX_REQUESTS it only forwards the specified number of requests to a child process before killing it and starting a new one. Problem solved.

Information on this behaviour can not be found in the PHP online manual, but rather at http://lxr.php.net/source/php-src/sapi/cgi/README.FastCGI.

Python everywhere: extending applications with Python

2009-03-31T16:33:00.000+02:00

Extending applications with Python: gimp, Evolution, Inkscape, Paint Shop Pro, [...]

Python everywhere: A Python Operating System called cleese

2009-03-31T16:31:00.000+02:00

Cleese....

Python everywhere: conficker scanner

2009-03-31T14:30:00.000+02:00

This article is the first in my new series "Python everywhere".

As this is the first article in this series I would like to explain what the series is all about.
As an avid Python user and developer I want to share my observations whenever I find Python applications doing not-so-unusual things, Python applications running on embedded devices. In the end I want to point out just what the name of this series suggests: Python is everywhere and can be used for everything.

So, straight ahead to the first issue: the conficker scanner.

When reading an article about a detection mechanism for the conficker worm on heise Security [german] I was myself wondering a few things, but wanted to give it a try. So I followed the link to the article Detecting Conficker, by Tillmann Werner. Before clicking the link I was wondering whether I could get this tool running on GNU/Linux using wine, or another method.

After downloading the ZIP file and unpacking it I thought I was dreaming. There were two Python files, along with a COPYING file.
So, even though before having a look at the code I wanted to know the COPYING conditions, and again I saw something unexpected: it's licensed under the GPLv3, great!

As there are some computers running a proprietary operating system from Redmond on this network I immediately gave it a shot. I started the script (scs.py), and after fulfilling its requirements (namely the impacket Python module) I ran it on the local network and it worked without any problems. No conficker found on this network, after all my flatmates have their systems secured - good.

So there you have another use-case for Python: detecting malware over the network.
Kudos should go to Tillmann Werner, not only for this piece of Python code, but also for his work on the honeynet project and, together with Felix Leder, the great analysis of conficker. Keep up the good work, and thanks for proving Python can also be used for this task.

Introducing pyttpd

2009-03-31T02:27:00.000+02:00

In this article I would like to inform you about my newest pet-project: pyttpd.

pyttpd is my effort of implementing a webserver in Python, with a focus on security (through privilege separation), extensibility and scalability.

I started this project because I was not entirely happy with the lack of flexibility and support for privilege separation by popular webservers. Whilst both lighttpd and Apache httpd provide means of running processes under different users these usually require hacks like suexec. Additionally I am somehow curious about how a fully-fledged webserver implemented in Python would perform compared to the mentioned daemons.

Security through extensive use of Privilege Separation

Whilst it is common for daemons to initially run as a privileged user and drop privileges as soon as possible it is possible to make more extensive use of setuid and friends.
pyttpd's design aims at creating one privileged process, which only binds to privileged ports and spawns subprocesses.
All subprocesses have specific tasks, such as routing between all processes, protocol-specific parsing of incoming requests and handling processing of those requests.
The point is that all these processes do not run as "www-data" or another common account, but that a logic separation takes place on a per-host basis. This means that if the webserver is hosting www.example.org and webapp.example.org those will be running under different system accounts, making it hard to interfere with each other. This method should also enable the use of MAC mechanisms such as SELinux or SMACK more efficiently.

The design choice of having separate processes for each vhost comes with another benefit: users (or customers) "owning" a vhost could potentionally be allowed to modify parts of the vhost's configuration (excluding UID, GID, and other security-relevant options) on their own.

Early status

Right now pyttpd is in a very early planning stage, with no code to show yet. I am still in the middle of the process of writing down all ideas that come to my mind, weeding some out and documenting the others.
So what do I have to show you then? Well, the the concept section of the documentation is online now and I am planning on extending it in the next few days and eventually start writing code rather sooner than later.

Your ideas...

...and opinions are what I am really interested in. If you are interested in this project I would love if you got involved in some way. Feel free to create tickets at pyttpd's project page if you have an idea you feel is worth adding or if one of my ideas is flawed, create a comment here or send me an email.

UPDATE: AdSense on freedom blog reloaded

2009-03-30T12:30:00.000+02:00

I just wanted to inform you that I am in the process of adding AdSense ads to this blog.
However, I am planning on having a one-ad-per-post policy, whilst not placing any ads on the front page.

More details on this topic will follow in the next few days.

UPDATE:

I have now integrated AdSense into this blog. As promised the front page does not contain any ads, but all other pages do. Ads are shown as a widget so they are not in-text and thus should not disturb you whilst reading.

python-argvalidate has hit Debian unstable

2009-03-29T13:32:00.000+02:00

I am proud to announce that python-argvalidate has hit Debian unstable yesterday.

This does not only mean that you can install argvalidate on Debian-based systems more easily now, but also that python-argvalidate has met the strict criteria of the Debian Free Software Guidelines, and as such has been confirmed to be Free Software.

Also, I wanted to let you know that I am maintaining the Debian package itself, which means that updates to python-argvalidate itself will be included in Debian as fast as possible, usually within two days.

How using proprietary software can affect system security

2009-03-29T13:17:00.000+02:00

There has been a lot of discussion on whether Free Software is more secure than proprietary software, but I have an additional argument that shows how the use of Free Software can improve system security.

Now you probably expect me to come up with a pure technical reason showing superiority of Free Software, but I am taking another path this time: let's talk about user trust.

Software updates on pure Free Software operating systems

Firstly, let's take a look at how security updates are deployed on pure Free Software operating systems.
All major GNU/Linux and *BSD distributions come with built-in update systems for all Free Software packages provided by the distribution. Once a security update to any piece of software is released you will get a notification saying updates are available and that you should install them and after doing so your system should be in a quite secure state.

Updater-applications of proprietary software

Okay, proprietary software does often come with its own update mechanisms, giving you pretty much the same functionality. But here is my point: users tend to block these updater-applications from accessing the network with personal firewalls and similar tools, which basically disables the updater and most likely will cause the system to be more vulnerable rather sooner than later.

But why do people even consider blocking an updater-application?

When I asked people why they blocked these applications a lot of reasons came up, but it usually boils down to lack of trust for proprietary software vendors. People seem not to trust software vendors for a variety of reasons. The most common reasons seem to be that they are either worried about the disclosure of private information or the lack of a license for a piece of software, combined with the first reason.

So people actually seem to be scared by what a piece proprietary software could do, and when not having their software licensed people seem to be even more scared. When I ask people why they are running these programs, even though they do not trust their vendors they usually shrug and I get replies that can be summed up as "it just works", "but I don't have a choice" and "I am used to software X".

The solution is Free Software

Free Software can be a solution to all these problems. The most important thing people should be aware of that they do not have to fear Free Software vendors. After all, when using Free Software you do have a valid license and you can, at least in theory, check exactly a program is doing to your system. Also, if you are not capable of doing such checks yourself, you can rest assured that other people are doing such checks and give back to the project in question, improving your favorite Free Software applications.

Free Software is a choice, and there is hardly any proprietary piece of software which cannot be replaced by a free equivalent. Free Software is my first choice, not only for me but for my family too, with great results so far.

Rest assured you can trust in what your software is doing, free yourself today - use Free Software.

A possible attack - what to do about this?

2009-03-26T13:06:00.000+01:00

Just as I wanted to start writing an article here and I entered the URL of this blog into my browser I got no response from the webserver, zero, nothing.
First I thought the PHP fastcgi process for this virtual host died, but a quick check on another virtual host suggested that something else was going on.

So I guessed the lighttpd process itself must be experiencing problems of some sort, but after doing a "netstat -nat" I knew what was going on:

tcp6       1      1 83.65.62.72:80          61.135.190.248:12474    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.234:39671    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.253:39211    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.234:55160    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.230:25836    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.231:16865    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.232:24266    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.240:38441    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.243:17726    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.241:38206    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.251:23892    LAST_ACK
tcp6       1      1 83.65.62.72:80          61.135.190.225:29675    LAST_ACK

Plus "a few" more of those. Now I'm not entirely sure whether it's just some systems misbehaving or actually an attack, but my feelings told me this could have been intentional after all.
I did a quick whois on one of those IP addresses and came up with the 61.135.0.0/16 network which is owned by China Network Communications Group Corporation.

As the connections were made from pretty much every host in that network I had two choices: sit it out or block it.

I came to the conclusion that blocking the entire subnet from connecting to this system, at least temporarily, might be a viable solution and so I did.
However, afterwards I am asking myself whether I really had to block an entire 16-Bit network, so I am asking you: how do you handle such situations usually?

python-argvalidate 0.9.0 released

2009-03-24T17:02:00.000+01:00

Even though I planned providing a release candidate first, which can be seen in the project's Mercurial changelog I have released python-argvalidate 0.9.0 today. Tarballs can be obtained from the Python Package Index (pypi), as usual.

So what's has changed compared to 0.8.2

Basically a lot of code has been rewritten or reviewed and modified. The most important change lies in argvalidate's internals though.
Instead of relying on Python internals, those are the func_* and func_code.* attributes of every function argvalidate uses another mechanism for obtaining the information about functions now, the inspect Module. In short this means that argvalidate should be more resistant to possible changes in the format of these attributes, because the inspect Module that ships with Python should be changed accordingly and it is unlikely that its public interface is going to break.

Which leads me straight to the next point regarding argvalidate: the public interface. The most visible change was the deprecation of the method_args, func_args and return_value decorators. These decorators are still present, but will be removed in one of the next upcoming releases of argvalidate. The functionality of method_args and func_args has been combined into the accepts decorator, whilst the returns decorator replaces return_value.

accepts now includes some magic for automatically determining whether a function or method has been decorated. Initially the plan was on using inspect's ismethod function, which did not work out as when the decorator is called for methods the method has not been bound to the class yet and looks like a simple function to ismethod. However, the magic added checks the first argument's name and decides that functions having either "self" or "cls" as their first parameter are methods, whilst everything else is a simple function.

There is more to say about accepts. Internally some checks which were done at every call to the decorated function in the past are now done only once, when the decorator is invoked. This should reduce the overhead argvalidate adds when the function (and thus the internal wrapper doing the checks) is called, but most likely will slow down decoration itself. However, decoration is only done once, whilst a function is likely to be called several times in its lifetime, which should give us an overall improvement of speed.

The documentation has been updated accordingly and can be found at the usual place.

What's up next?

There is no real further roadmap for argvalidate right now. No issues were left open, except for Python 3 testing & support. Further releases are going to be made when needed, that is when you submit a bug report and it is fixed.

Presented in H^H^H^H^HIPv6

2009-03-17T12:49:00.000+01:00

I just wanted to let you know that this blog (actually all webpages I am hosting) are now accessiable via IPv6. Additionally, my mail-server now also accepts IPv6 SMTP and IMAP connections, allowing communication with the IPv6-world.

The setup uses SiXXs as tunnelbroker, with AMIS being the SiXXs PoP in use.
If you experience any problems with the services I am providing via IPv6, please let me know, either via a comment to this article or an email to ipv6@sp-its.at.

Freedom blog reloaded launch

2009-03-17T12:31:00.000+01:00

Welcome to my new blog, "freedom blog reloaded".

Now with this first article I would like to elaborate on the name of the blog, the purpose and what you are likely to find here in the future.

Okay, let's start straight ahead with the name of the blog. Freedom in the blog's name refers to Free Software, which is going to be the main topic of the articles you will find here.
I would like to keep you informed about my involvement in the Free Software community and hopefully provide you with some useful information when it comes to configuring and running Free Software.

Now you might still ask what the "reloaded" part in the blog's name is about. Well, I have done some blogging in the past, but due to various reasons didn't have the time to provide my readers with a constant flow of articles, but this should change now. I am planning on regularly keeping you informed.

On to the last thing I wanted to write about: the kind of articles you are likely to find here in the future.
I am planning on writing posts on development in the Free Software community, updates to the Debian GNU/Linux packages I either maintain or co-maintain, the projects I am working on and last but not least some tips and tricks when it comes to day-to-day operation.

Lastly, as this is a blog dedicated to Free Software it's a good idea to let you know that this blog is being run on a Free Software stack completely and I am using Free Software only to write articles.
The setup is as follows: Running on a Debian GNU/Linux system is lighttpd, my webserver of choice, and builds, along with PHP5 and MySQL, the base for running Wordpress, a blogging system written in PHP.
For writing articles I am using, guess what, a browser, namely Iceweasel (also known as Firefox to non-Debian users), running on my Debian GNU/Linux workstation.

I guess that's it for now. As a last note I would like to point out that even though comments have been disabled for this article I will enable them for all posts where discussion makes sense.

-- Stephan

How Email encryption for the broader public could be realized

2008-10-16T12:27:00.000+02:00

After reading the Every Email In UK To Be Monitored article and its comments over at Slashdot I once again felt like encrypting each and every Email I send using GPG/PGP. Now for this encryption to work the person I am sending a message to would need to have GPG/PGP set up too. A lot of technical-minded people already have this set up, but I can not expect everyone to be using encryption.

The reason for not everyone using GPG/PGP for encrypting their emails might be that, even though GPG/PGP have become a lot more usable for the end-user in the last few years, these programs are probably still too technical and thus hard to understand for non-technical users.

This is when I thought a little about how people could be made using public key encryption for E-Mails. After a bit of brain-storming an idea came to my mind, an idea I would like to present you with.

Basic idea

What about creating a program acting as both SMTP and POP3/IMAP proxy server that included all the logic to do encryption and would encrypt/decrypt messages transparently?
If this logic was moved out of Email clients we could get a solution working universally for each and every Email client out there.

How this could work

Imagine you sending an email to someone you've never sent an email to. You write the message in your Email client as you are used to and hit the send button. Now, instead of connecting to your SMTP server the E-Mail client would connect to the Email proxy program and submit the message there.

At this point the program would check the email sender and recipient. If the sender does have a public/private key pair and the recipient's public key is known the program would prompt you for the passphrase to your encryption key. After entering the passphrase and hitting a button (send, sign, encrypt, I guess you can think of a more appropriate name) again the message would be encrypted and then forwarded to your SMTP server.

On the other hand, if the public key of the recipient is not known (and cannot be fetched off key servers) the program could send a message informing the recipient that you wanted to encrypt your email, but were unable to do so, explain that this program exists, where to get it from, how to set it up, why encryption is important, and so on. I can imagine having a hard-fail mode, sending only this message and a soft-fail mode, attaching or including the automatically generated message somehow (attach it, inline it, etc.) to the original message. Either way, the generated message should be cryptographically signed.

Receiving mail would work the other way around. The proxy would try to fetch messages off all configured IMAP/POP3 servers on its own, check if they are signed. If a signed message arrives the public key should be, if not already done, be imported into the local keyring. As for encrypted messages this should happen the same way, plus decrypting the message.
The Email client would connect to the IMAP/POP3 proxy server and fetch (the decrypted) messages from it. Both unencrypted and unsigned messages should be marked somehow (think subject re-writing here and maybe adding an X- header). However, no automatic sending of emails should happen when receiving messages as the From header could be forged (spam anyone?).

Features of the program

The program I have in mind should include the following features:

GPG key management (creating, distribution to keyservers, etc).

Automatic encryption/decryption and signing/checking signatures.

Non-technical, so everyone can use it.

Support multiple IMAP/POP3 and SMTP servers, so it can act as a central point for storing all Emails a user could receive.

Cross-platform functionality (Java? Python?)

Free Software

Plans

I would love to implement this program, but fear that this could be way too much work for a single person. If you are interested in helping with the implementation or simply have any comments feel free to either drop me an email at blog at sp dot or dot at or use the blog's comment function.

I hope I did explain my idea clear enough and did not miss anything.

Happy hacking!

EuroparlTV for everyone? No, only for users of proprietary software!

2008-09-18T14:19:00.000+02:00

The European Parliament (EP) has just recently started a new service: EuroparlTV. A web-TV service which should give citizens of the European Union (actually everyone around the world) a way to inform themselves about how the EP works, what it does, and so on.

After I first read these news over at heise (german) I was impressed, but started to fear that yet again some sort of government has invested in proprietary software and is able to bring its services only to users of such software. Seconds later my fears became reality.

EuroparlTV seems to work only for users of either Adobe's proprietary Flash player (via the proprietary Adobe Flash file format) or users of Microsoft's Windows Media Player (via the proprietary WMV file format).

What this means to an open web, that is usable for everyone, should be clear.

Basically this is a service all citizens of the European Union pay for, but some cannot use. Is this really how governments (and the EP is some sort of government) should treat their citizens? Rather not.

On the one hand the European Commission is fighting vendor lock-in and monopoles, but on the other hand it directly helps these vendors by creating such services. Not a smart move in my opinion, neither is it understandable.

What I am asking myself though is why the EP was unable to create such a service, which itself could be quite interesting, without having all users of that service use proprietary software?
Is it so hard to deliver the service in a free (as in freedom), standardized format?
I will let answering these questions to you, but keep in mind that there are alternatives to this whole proprietary mess, like Ogg, which are completly free.

Personally I am pretty disappointed by this move. However, I hope that I at least informed people that there is a problem with EuroparlTV.
Putting it simple and short this way the EP does a great deal with helping vendor lock-in whilst fighting the freedom of its own citizens. Even though it should be the other way round.

sptest - a Python unittest extension

2008-09-10T14:06:00.000+02:00

Even though this is meant to be an introduction to sptest, I want to start off by letting you know why I wrote this extension to the Python unittest module.

I am currently working on a (still private) project that uses Python's unittest module and the underlying framework. Even though unittest is a great utility for creating unit tests I found that the output it generates is unusable for me. I wanted something different though. Maybe a bit more aesthetic than the simple command line output unittest provides.

So I started off writing a class extending unittest.TestResult to fit my needs. I soon realized that interfacing with this part of unittest is not as easy as it could be, but I still continued to write my class.
After two hours of hacking I noticed that this class had become a monster. It was huge and I felt uncomfortable having such a huge class lying around somewhere in a "runtests.py" file for the only reason of having that pretty output.

This was the point when I decided to move all that code into a separate project and try to come up with a more intuitive API. This was the second when sptest was born, about 5 hours ago.

What I did come up with is a small Python module that makes customizing the way unit test results are presented (or stored) easier. It currently includes two output handler classes. One providing fancy CLI output on ANSI terminals and the other one providing XML output.

Additional output handler classes could store the result of the unit tests in a database or send it to a central point on the network, but implementing that is up to someone else, for now.

Running unit tests with sptest is as simple as calling:

sptest.TestMain(TestSuite).run()

By default the FancyCLIOutput handler class will be invoked and you will see why the handler is called the way it is immediatly.

In order to generate an XML file containing the test results one just has to modify the call to sptest to look like this:

sptest.TestMain(TestSuite, output_class=sptest.output.XMLOutput).run()

sptest also provides support for preparation and cleanup functions. The only thing you have to do is define these functions and adjust the arguments passed to TestMain accordingly.

Most of the code is already documented and a doxygen configuration file for generating the html documentation comes with the code. Also, two examples are included that show how to use sptest.

UPDATE: Google Chrome: Good or evil? -- GOOD!

2008-09-02T18:54:00.000+02:00

UPDATE: You can find the update to this article at its bottom.

Even though Google's slogan is "don't be evil" I am not entirely sure whether this also applies to their newest development: the Google Chrome browser.

The announcement over at the Official Google Blog tells us that Google is about to release a Free Software-based browser. When I first read the announcement I wasn't too impressed reading that Google has actually built a browser, this was logical and I have been expecting this move for years. Also, reading that they based their browser on Free Software didn't impress me too much either, but then I found the comic.

The comic contains a lot of information about the browser's architecture and I like the design. It makes perfectly sense, even though it could create some memory and processing overhead, but don't all major browsers consume "quite some" ressources? So, from a technical point of view, the browser sounds great, but there is a huge downside too.

The product announcement says that the browser is not only built upon Free Software, but is Free Software itself. Now, this sounds good, but then I had to read this:

This is just the beginning -- Google Chrome is far from done. We're releasing this beta for Windows to start the broader discussion and hear from you as quickly as possible. We're hard at work building versions for Mac and Linux too, and will continue to make it even faster and more robust.

I don't want to start nit-picking on the use of the term "Linux" for describing the GNU/Linux operating system there, even though I have to mention this fact.

What really bothers me is that it seems as if a binary-only release for Windows is being prepared, and only this binary version. In my opinion this is bad. I would rather have liked reading "a binary beta version for Windows will be made available along with the source code licensed under the terms of the <insert your favourite Free Software license here>".
Why? Because this way people could start tinkering with the code and thus help making a GNU/Linux version available sooner. Not seeing the code released makes the "Free Software" promise sound void.

Though nothing has happened yet. Google has merely announced the upcoming release of Google Chrome. No details have been made available whether the code will be released along with the Windows binary, but I fear we won't be getting hold of the code for a while.

This leads me to the title of this article: Is Google Chrome good or evil?
Well, if Google keeps the promise to release Chrome under a Free Software license and does so rather sooner than later I believe Google Chrome should not only be called "good". It would then qualify as a real alternative to Mozilla Firefox and could even be superior to Firefox.
On the other hand, if Google does not release the code timely, releases the code under a proprietary license or does not release the code at all Chrome could and possibly should be tagged "evil".

Personally I am awaiting the release of Google Chrome. I would like to test it, see the code, maybe dig a bit into it and possibly make it my browser-of-choice. The reason for this is quite simple: I am tech-savvy and the technology used in Google Chrome sounds more than just interesting, but could actually be a step forward for the web. Both in increased usability for the user and the use of Free Software and Free Standards as a way to help the web evolve. If Google doesn't keep the Free Software promise though, expect me not to ever though that evil beast.

UPDATE (September 3, 2008 at 7:44am CET):

Now that Chrome has been released Google apparently did also release the source code to Chrome, Chromium. The chromium project page can be found here, the Google Chrome home page here.
Now it seems as if Google did make Chromium a Free Software browser (seems because I have not yet come around to downloading the tarball and checking the contents, but I do believe it actually is Free Software and for me there is no reason not to believe that anymore).

I am more than just happy with this because, as I pointed out in this article already, Google Chrome or Chromium does have an interesting architecture and should, in my opinion, be embraced by the Free Software community. The reason I am happy is not only the fact that it is Free Software, but rather that a company like Google does release a lot Free Software these days and personally I hope other companies will start following this example soon. Thanks Google for taking this step!

So to make it short: Google Chrome? Not evil, good!

Now a short word to the commenters of this article: Most comments have been helpful and I really appreciated them. Sorry that an update to this article took so long, but I'm living in Europe and was asleep while all the things you mentioned have happened.

Autoconf and Python: checking for modules

2008-08-31T13:04:00.000+02:00

I am currently writing a Python application that makes use of GNU Autotools as build system and noticed that determining whether a specific Python module is installed is not that easy and no usable Autoconf macro exists. So I came up with my own solution, which I would like to share with you.

The AC_CHECK_PYTHON_MODULE macro takes two arguments: The module name and optionally the variable name holding version information. This way it is not only possible to determine whether a module is installed (ie. loads in Python) on the current system, but also retrieve version information from that module.

The following examples checks whether the Crypto module is installed and retrieves its version information from Crypto.__version__:

AC_CHECK_PYTHON_MODULE(Crypto, __version__)

The macro itself does never report and error, but rather only a found/not found result. Error checking is up to the user and can be done via these two Autoconf variables:

PYTHON_<MODULE_NAME>

PYTHON_<MODULE_NAME>_VERSION

PYTHON_<MODULE_NAME> is set to "1" if the module is present and "0" if not present.
PYTHON_<MODULE_NAME>_VERSION is only set when the version variable argument has been set and contains the version information of the module, if the module been found. If the module is not present this variable is also set to "0".

The version variable argument is optional as I wrote, so the following invocation works too and only checks whether the distutils module is present:

AC_CHECK_PYTHON_MODULE(distutils)

As I wrote earlier in this article I would like to share this macro with you. You can download it here.

Debian GNU/Linux 5.0 ("lenny") on a Samsung P55-Pro T8100 Sevesh

2008-08-22T11:28:00.000+02:00

I have recently bought a new laptop, a Samsung P55-Pro T8100 Sevesh. As I was not able to find an installation report for this model anywhere on the internet I thought writing one myself is a good idea. This way people interested in getting this laptop or installing GNU/Linux on it can get some information.

The article covers both the hardware configuration of the laptop itself, a list of which features of the laptop do work and which don't (do not be afraid, most things work perfectly well out of the box) and finally a short installation report.

First of all, let's have a look at the hardware configuration of this laptop:

Intel Core 2 Duo T8100 CPU (2.1 GHz)

2GiB DDR2 RAM (PC2-5300 - 667MHz)

Intel GM965 chipset with integrated Intel GMA X3100 graphics adapter

250GiB HDD

SXGA+ display with a resolution of 1400x1050

Intel PRO/Wireless 3945ABG WiFi adapter

Intel 82566MC NIC

HD Audio Codec, ALC262 sound adapter

AuthenTec AES1600 fingerprint reader

Infineon TPM module

Ricoh cardbus bridge (RL5c476 II) plus cardreaders and IEEE1394 controller

One cardbus (PCMCIA II) and one Express Card/54 slot

Now on to the list of what does and what doesn't work with GNU/Linux.

Intel GMA X3100 graphics adapter

Works out of the box. Full resolution is possible without a hack, VGA out works out of the box in both mirror and extended desktop mode.
NO xorg.conf modifications are needed in this setup, everything works perfectly well with a nearly empty xorg.conf!
The only thing I had to modify was making the virtual display a bit bigger so that extended desktop mode works with an external monitor having a resolution of 1680x1050 pixels.

Intel 82566MC NIC

Works out of the box, no further configuration needed.

Intel PRO/Wireless 3945ABG WiFi adapter

Works with the iwl3945 driver, however, it requires something Intel calls "ucode", a proprietary firmware. Without this piece of firmware the card does not work. If you want to WiFi without the need for proprietary software (the ucode) you will have to go for a USB, PCMCIA or Express Card/54 WiFi adapter.

HD Audio Codec, ALC262 sound adapter

Works out of the box.

AuthenTec AES1600 fingerprint reader

The fingerprint reader is said to be working with fprint, which I did not test yet though. Expect an update sometime soon.

Infineon TPM module

Not tested.

INSTALLATION REPORT

Basically the Debian GNU/Linux 5.0 installation went smoothly using the beta2 netinstaller image. The system booted from cdrom and the installation process worked fine.
After rebooting into the new system however the system froze. No response, nothing. The last message on the screen suggested that the ACPI video module is the problem.
After rebooting using init=/bin/sh as boot argument I modified /etc/modprobe.d/blacklist and added the following line:

blacklist video

This is only a workaround for the real problem. The bug is present in Linux 2.6.25 and Linux 2.6.26. A bug report has been filed (here). I will update this page as soon as the problem has been resolved.

There is another thing which doesn't seem to work. However, this could be (and likely is) related to the broken ACPI video kernel module: adjusting the display brightness.
On AC power the system boots with maximum brightness, which cannot be adjusted. Unplugging the AC adapter lowers the brightness.
When running on battery one can use the "brightness up" key combination to switch to maximum brightness, however, this cannot be undone.

CONCLUSION

The laptop is not only usable under GNU/Linux but most hardware works, even out of the box. The only real problem is the broken ACPI video module, which hopefully gets fixed soon.
I hope this article helps those who would like to get one of these laptops, but are not sure of its GNU/Linux compatibility, just like I was.

Is trying to fix (E)SMTP really worth it? [part 2 - infrastructure]

2008-07-03T13:33:00.000+02:00

[digg=http://digg.com/security/Is_trying_to_fix_E_SMTP_really_worth_it_part_2]This article is the second in my series about the flaws of (E)SMTP, the whole Internet mail infrastructure and how it could possibly be fixed. The main focus of this part is a new approach to the infrastructure which should help making emailing more secure, reliable and less spam-prone.

The first article can be found here and points out flaws and problems in the current systems.

Before going into detail about how the infrastructure could look like I would like to point out the goals of my proposal:

security through end-to-end encryption

security through sender and server authentication

integrity of message contents

built-in load-balancing support

getting rid of email forwards

These five major points should be covered directly by a new infrastructure and should be mandatory. There is no point in making any of these optional as the rest of this article should point out.

security through end-to-end encryption

Even though both SSL and TLS support exist for (E)SMTP these features are optional. In fact this means that it is possible that even though one submits his or her email over a secure channel the message could be transferred in plain-text somewhere on the way to its destination.
This enables an attacker to snoop at your message somewhere along its way. Whilst some people believe this is okay I strongly oppose to anyone being able to read either my private or business emails.

The solution to this problem is end-to-end encryption. The new infrastructure should make encryption of all message exchanged mandatory and further provide a way of encrypting the message contents. This way only the intended recipient can actually read the message (as in not even a server administrator having direct access to a user's mailbox). End-to-end encryption of the communication channels should be done by using TLS for all communication between all clients and servers and for server-to-server communication.

Encrypting of the message payload could be done in a similar (if not even the same) way OpenPGP (RFC 4880) works.

security through sender and server authentication

The next feature a possible SMTP successor should provide is sender and server authentication. As TLS should be mandatory for the implementation the easiest way to achieve this is using a public key infrastructure. This could then in turn be used for multiple things, including message integrity checking, encryption of message contents, authentication of the sender and authentication of the server.

Integrating a public key infrastructure could be done by having special DNS (maybe TXT) records that contain the address of key servers. These key servers would store not only a domain-root certificate which would allow user and server authentication but also all user and server certificates themselves.

A receiving server could then check the sending domain's key server for both the domain-root certificate and the sending server and thus verify that the message is legitimate and actually originated from the specified domain.

Sender authentication works together with message integrity. Basically the receiving server opens the message, gets the client's message signature from the message, and asks the sending domain's key server for the public key of the sender. The receiving server then checks the signature and this way verifies the sender.

integrity of message contents

Integrity checking is closely related to sender verification. As the receiving server checks the sender's message signature in the sender-verification process the message is automatically checked for integrity too.

built-in load-balancing support

Load-balancing is also closely related to the PKI approach. The sending server could use the receiving domain's key server to locate the server to send the message to. This way load-balancing of receiving servers can easily be implemented. Furthermore load-balancing of multiple key servers for a single domain is possible using DNS round-robin records.

getting rid of email forwards

Forwards can also be gotten rid of by using the receiving domain's key server similarily to the load-balancing approach. Instead of pointing the sender to a domain-local receiving server the key server could simply point the sender to another domain's receiving server. This way the message would not really be forwarded or relayed anymore but rather a pointer to where the message should be stored could be provided.

Putting everything together

Making all the mentioned features mandatory for a possible successor of SMTP should make users benefit in a few ways. Firstly, users could rely on both then integrity of the message, that the sender actually the person he or she pretends to be and the fact that snooping on the contents of the messages they send is hard to impossible.
Furthermore this infrastructure should make sending SPAM messages a lot harder as domains for sending spam would have to be bought, DNS servers and key servers would need to be operated and blocking unwanted messages could be as easy as blocking either a domain or a single user using the information provided through their message signature.

ISPs would benefit from the built-in load-balancing mechanisms and the mailbox alias feature (forwarder). Whilst the load-balancing technique simplifies set-up and operating of a load-balanced infrastructure the mailbox alias feature should help cutting down on traffic generated by email forwarders.

Please be aware that I intentionally left out all implementation specific details, such as the message exchange protocols. More technical aspects of a possible implementation are to be covered in the next parts of this series. As always, comments are highly appreciated.

Status update

2008-06-23T13:38:00.000+02:00

It has been quite a while since I last wrote an article and published it here.

It's not like I got tired of blogging. The reason why there hasn't been an update for such a long time is that I was doing my final exams in the past two months.

After passing my exams on Friday I should have time to write some articles again, so watch out for new articles here.

Why are hardware manufacturers keeping specs to themselves?

2008-04-06T16:13:00.000+02:00

This is one question I have been interested in ever since I started using GNU/Linux.

Just think about it for a moment. About 20 years ago you got specifications for pretty much every piece of hardware you bought. You were given exact instructions on how to use the hardware you just bought, not only how to install it. Things have changed since then.

If you buy any piece of hardware today you actually have to expect not to get any documentation on how to "talk" to your new toy. You are only given a CD (sometimes even only a link to a homepage) containing drivers for a few specific operating systems, usually only Microsoft Windows.

Now I am no driver hacker and so I probably wouldn't be able to implement a driver for anything on my own anyways, but the Free Software community would largely benefit from hardware documentation, as there are a lot of capable driver hackers out there.

This is not a problem that only affects the Free Software community though. There are a lot of pieces of hardware which do not work on recent proprietary operating systems anymore due to lack of support by its manufacturers.
At least this problem would not exist for Free Software operating systems, such as GNU/Linux, if hardware makers would publish documentation of their hardware. The people still using devices which are well beyond their end-of-life could implement drivers on their own, not being dependent on anyone.

What I am really wondering about in this case is why hardware companies are unable to coin standards for accessing devices of the same class. It works perfectly well for USB (take USB mass storage devices as an example) and I do not understand why there can't be standardized interfaces to other hardware, such as network adapters, as well. On a very-low level these standardized interfaces do work. Just think of PCI, PCI Express or AGP.

Actually, if you think about this for a few more seconds you should realize one thing: Having standardized interfaces for devices of the same class would cut a lot of costs for hardware makers. Why? Oh well, if they design a brand new networking chip and still implement the given standard there would be no need of writing a new driver. Wait, there would be no need for per-device drivers at all. Implementing a common driver that accesses the standardized interface would be enough, for a whole range of devices.

So what am I asking of hardware makers? I would love to see companies creating devices of the same class to get together, create standardized interfaces, publish them and implement them in their new devices.
I know, this is not likely to happen anytime soon, so a more realistic approach is asking for Free Software drivers and/or documentation.

Personally I have stopped buying hardware which "works" with GNU/Linux, I have come to the point where I try only to buy hardware which either comes with Free Software drivers from the manufacturer or documentation which allows implementation of Free Software drivers.
This is probably the best way of showing these companies what you demand: Freedom.

Free Software Supporter

2008-03-28T16:13:00.000+01:00

I was quite stunned when I noticed that the Free Software Foundation (FSF) has recently started a new monthly-published newsletter, called the Free Software Supporter.

The reason I was amazed is not the fact that the FSF is now publishing such a newsletter, but rather the fact that I did not hear about that yet. Basically, the Supporter is about informating the Free Software enthusiasts about recent happenings and the work of the FSF, the GNU project and the global Free Software community.

It seems as if I am not the only person that is excited about the supporter, as Joshua Gay, who apparently is writing the Supporter, also seems to like it, as he writes in a blog post:

I hope that you enjoy the Supporter. I am looking forward to reflecting each month upon the work of the FSF, the GNU project, and the global free software community. I only hope that the number of highlights I add each month will continue to grow as quickly as the community is growing. In either case, we hope to keep it short and we hope to keep you informed.

You can sign up to receive the Supporter via email on a monthly basis at http://lists.gnu.org/mailman/listinfo/info-fsf and you can read the first issue online at http://lists.gnu.org/archive/html/info-fsf/2008-03/msg00000.html.

Also, if the Supporter looks like an interesting read to you, you may as well enjoy the monthly newsletter the FSF Europe publishes. The FSFE Newsletter can either be read online or you can sign up for the FSF Europe press-release mailing list.

Personally I believe both newsletters are worth reading and give you a great overview of what has happened in the past month, what is going to happen and the work done by the FSF and FSF Europe.

Is trying to fix (E)SMTP really worth it? [part 1]

2008-03-27T20:36:00.000+01:00

[digg=http://digg.com/security/Is_trying_to_fix_E_SMTP_really_worth_it_part_1]
This one question has been in my mind for quite some time already. I mean, everyone uses SMTP (knowingly or not) when sending out emails and everyone sending emails also knows what SPAM is and receives SPAM messages.

However, few know how old SMTP actually is, and that, even though it serves everyone well, it has been designed in a time when everyone was thinking of Spam as canned meat. Back in 1982 SMTP was a great achievement and a lot of kudos should go to its creators, but now, in 2008, SMTP has become more of a liability than a great tool.

Originally, I wanted to write a single article covering all shortcomings of SMTP and possible solutions to these problems, but while writing the article a lot of text came up, so this is the first of two articles I am going to write on this topic. The first part is about the problems with SMTP and how fix-ups for SMTP are, even though they do work to some extent, a proper solutions to today's issues.

Due to the way SMTP was designed and the way the Internet was back then it is prone to various things, like SPAM messages, sender spoofing, data manipulation and so forth. A few attempts have been made at fixing some of the shortcomings of SMTP, like ESMTPA (SMTP-AUTH) or SPF, Callback Verification, and DKIM, but none of them has really fixed all problems that exist and all of these modifications are in my opinion mere workarounds. Let us have a look at why both SPF and DKIM fail to fix the all problems SMTP has right now.

SPF

First let's have a quick look at how SPF works: When an email is received a special TXT DNS record at the sender's domain is used to verify that the sending computer (using its IP address) of an email is actually allowed to send email for a domain. A great mechanism that in theory would work perfectly well. Reality is a bit different though.

There are some domains which are actually using SPF and do have valid SPF records on their DNS servers. However, those are only some of the millions of domains on the Internet. How should one treat emails coming from a domain without SPF record? The messages could be real, non-SPAM messages, that should be delivered, but on the other hand these could be SPAM messages. Also, the more people start using SPF, the more likely it becomes that spammers are simply going to use sender-domains which do not have SPF records.
Also, there are some organizations that have domains with improperly configured SPF records and there are even well-known ones, such as Technorati (I covered this in one of my articles). So one cannot even trust SPF records and valuable messages could be lost if a mail server is configured to drop all messages for which SPF authentication fails.

And there is a third problem: Sending emails from other places than your default one (office, home, etc.) and ISPs not allowing external users to use their SMTP servers (even not with authentication). A good example of this would be Austrian ISP UPC (their SMTP server tells me that the AUTH extension was not advertised, even though it was; long story short, I cannot log in from outside) and I am quite sure there are a lot of others.

And I can come up with yet another problem: What about email relaying? Think about downloading all messages from all your email accounts into a single one, using fetchmail for example. This makes SPF useless, as no checks can be done anymore, due to the sending system's IP address not being the original sender. If one assumes that every mail server uses SPF this is not a problem, but I like doing my checks on my server rather than relying on some other server.

Maybe there are even more problems with SPF, such as what to do when an email is received from a nonexistant domain or when there is a temporary DNS failure on either side, but the ones listed above are those I am confronted with most often.

Callback Verification

On to the next topic. Callback verification is a simple method used by mail servers to try verifying that the sender actually exists. Whilst this works for some SPAM messages which use non-existent senders, it does not help much as soon as the sender address does exist, and it does not even matter if the message was actually sent by the user owning the address. I guess there is nothing to add, even though it is a nice method to get rid of some spam it does not help with a lot of such messages.

DKIM

DKIM, or DomainKeys Identified Mail (originally named DomainKeys) is a method that is not meant to prevent abuse (such as SPAM), but rather to make tracking abuse easier. It works by the sender (usually the MSA of the sender on behalf of him) adding another header, "DKIM-Signature", which contains a cryptographic signature of the message body. The signature is generated using public-key cryptography, where the public key is stored in a DNS TXT record and can thus be used by the receiving end to verify that the message contents have not been tampered with during transport and that the mail actually originated from that domain.

This method, even though being one of the most advanced ones today, is prone to replay attacks and does not protect from tampering with message headers. In short this means that even though the message body cannot be modified without the receiving end detecting the modification, the headers can, and thus the message can be redirected. Also, it is possible to intercept the transmission of a message, generate a thousand messages with the same content but a different recipient and this way flood a mailbox with a message that would stand DKIM verification.

The email relaying problem that's present in SPF is not a problem here anymore, but the mobile-mail, the nonexistent domain and the DKIM-not-in-use problem still exist.
Also, DKIM seems not to be used by a lot of email servers on the Internet. Thinking about it for a second I can just come up with two names of well-known organizations using DKIM: Google and Yahoo.

The message format

The next part of this article is about the SMTP message format. This part is not directly related to the SPAM problem, but should provide you with some more information that verifies that SMTP is outdated nowadays.

RFC822 messages (or emails) usually consist of two parts: a message header and a message body. Originally these messages were designed to contain 7bit-encoded ASCII data, which is plain text. This means that there were only 128 different characters which could be transferred via email, without support of special characters, like German "Umlaut" characters. A solution has then be developed, not only to support special characters in emails, but also to support transferring of binary data (such as images).

MIME is the name of this solution, and it enables every one of us to send binary attachments and special characters via email today. MIME allows the email client to include more than just 7-bit plain text messages, including attachments. This is achieved by special header, "MIME-Version", which indicates that the contents of a message are MIME encoded. This header is then followed by a "Content-Type" header, identifying the type of content. For simple messages just consisting of a message body this would be "text/plain", telling the client that there is just text in the mail.

However, how can emails then consist of both text and attachments? Well, there is a special value for the "Content-Type" header: "multipart/mixed". This one indicates that there are several parts of a message, and every part comes with a separate "Content-Type" header. This way contents of a message can be organized in a tree, for example, containing the message body and a forwarded message.

An attachment is added by specifying an additional part of the message, usually with a "Content-Transfer-Encoding: base64" header, that says that the data has been base64 encoded. This way binary data can be represented using 7-bit ASCII.

But what does that mean? First of all, even though a message is split into several parts, there is only one body. Now if you are downloading a message via POP3 for example, there is no way of only downloading the actual text. You always need to download the whole message. Everyone knows this situation: You are downloading a message and have to wait for all attachments to be downloaded, even though you might not be interested in those attachments at all.

Also, encoding binary data using base64 creates a lot of overhead, as every byte (which can have 256 different values and corresponds to a single letter of text in 7-bit ASCII) is represented using only 64 possible values. Talking numbers here this means that messages encoded using base64 are usually 137% the size of the data they contain.

Conclusion

I hope that I have shown you what the problems with SMTP are right now. SMTP initially was designed to transport only text and had no way of verifying either the sender of a message or the integrity of data. Some workarounds have been created to get rid of these issues, but even though some helped a lot, none has really fixed any of those problems.

Also, one should never forget how much time and money has been used to try fixing SMTP, whilst a lot less money might have been sufficient for creating something new, something better, something that is built for the needs of the Internet today, and not for the needs of the Internet back in 1982.

Personally I believe that the days of SMTP are long over and that there is need of a proper replacement. I do understand that SMTP and the current email infrastructure are still in use because an infrastructure exists, but SMTP really deserves being retired, after serving us pretty well for more than 25 years.

The next part in this series of articles will be about what my idea of a successor of SMTP and the whole email infrastructure is, what it could look like and also how it could work. So stay tuned.

How to reject mails containing OOXML attachments using Exim4

2008-03-27T12:33:00.000+01:00

I finally did it. I modified my Exim's configuration to reject any mail with an OOXML attachment (ie. docx, pptx, xlsx).

There are two main reasons for this step. First of all I am not able to open these files and I believe I will not be able to do so and get them properly rendered anytime soon. Secondly, people using the new Microsoft Office suite seem to be ignorant enough to think everyone is able to view those files, which is not the case.

I am trying to make one point here:
People sending emails to other people should always send files in internationally standardized formats (open formats), such as ODF or PDF, so that everyone is able to open them and use the attachments. Also, I am trying to make people sending out emails in those formats aware of the fact that not everyone can open them, not everyone wants to invest a lot of money in new applications and that some people generally prefer Free Software and that there is no way of using those files using Free Software right now.

Enough for the introduction, I wanted to explain how to achieve this behavior using Exim4:

deny message = Message contains attachment of unwanted type ($found_extension)
demime = docx:pptx:xlsx

Putting this snippet in the acl_check_content section of your exim4.conf should do the trick.

Oh, and while I am at it, you can easily use this snippet to drop mails with other attachments, based on the file extension.
For example, in order to reject all mails containing WMV files just use demime = wmv.

Note that this snippets checks for a specified file extension instead of a MIME type. People still can get mails through in those formats if they modify the file extension, so do not use this method as a security measure.

SFLC now also providing services to for-profit clients

2008-03-26T23:12:00.000+01:00

The Software Freedom Law Center, known for providing pro bono legal assistance to Free Software projects, announced the formation of Moglen Ravicher LLC, a law firm also providing services to for-profit clients.

"We are pleased to extend the services of the Software Freedom Law Center to companies that support software freedom," said Eben Moglen, founding director of SFLC.

Moglen Ravicher LLC is fully owned by the Software Freedom Law Center, and all profits will go to support SFLC's operations. Clients of Moglen Ravicher LLC will receive legal counsel from the same attorneys that staff the Software Freedom Law Center.

This not only means that companies are now able to get legal assistance on Free Software matters from the SFLC, but also that the center found a way of helping its own funding.

It also seems as if the first for-profit client is OpenNMS:

An initial client of Moglen Ravicher LLC is OpenNMS, an open source enterprise grade network management platform. OpenNMS has retained the firm for representation regarding violations of the GNU General Public License (GPL).

For more information see the homepage of the SFLC and the news entry announcing this step.

Happy Document Freedom Day!

2008-03-26T21:05:00.000+01:00

Just in case you do not know yet: today is Document Freedom Day.

Today is Document Freedom Day: Roughly 200 teams from more than 60 countries worldwide are organising local activities to raise awareness for Document Freedom and Open Standards.

What does this mean for mean personally? Less than one would expect. I have been advocating the use of Open Document formats (such as ODF) for the past two years already, and try to do so whenever possible.

People react very differntly when I raise this issue. Some appreciate being informed that there are Open Document formats, which guarantee interoperability with everyone, but others tend to tell me "everyone uses [Microsoft] Office, isn't that format a standard?". The answer is always the same: NO.

Neither the old proprietary Microsoft Office format, nor the new format, OOXML are standards in my opinion and here is why:

The old format is not documented at all, and no international standards body, such as the ISO, have ever made this format a standard.

The new format, OOXML, which is in the news quite often lately, is being pushed to be made an ISO standard. People often think that, as documentation (which is said to be of poor quality) is available, making this format an international standard would be a good thing.
I am afraid I have to say NO once again here. There are too many references to the old proprietary format, which is a huge no-go for something that should become an international standard.

Also, there already is an international standard for office documents, ODF. In my opinion there is no point in having two separate standards for the same thing and the chance of such a situation causing a lot of havoc is quite good.

So, personally I have to say that I quite often suggested people to switch to OpenOffice.Org lately, instead of buying Microsoft's latest Office suite. Document Freedom and the use of Free Software are not my main arguments lately, but rather that people switching to OpenOffice.Org now do not have to learn how to use a new user-interface. People are lazy, and this argument works perfectly.

And there is yet another point for using Open Standards in IT:
Think of the Internet and where it would be without Open Standards (and also Free Software). Think of how everything on the Internet would work together. Think of one browser supporting only its own network protocol (which of course would be proprietary) and other browsers only supporting theirs. The Internet would not be what it is today without Open Standards and guaranteed interoperability.

More information about the Document Freedom day can be found in the last news entry over at documentfreedom.org.

Less spam again

2008-03-26T17:26:00.000+01:00

I found a solution to the problem last described in this article.

To sum the problem I was experiencing up: My anti-spam system (namely Spamassassin) did not detect spam mails anymore.

Now here is the reason it did not: After some more investigation of the problem I noticed that spam emails were received via a local connection (forwarded from fetchmail). However, one of my Exim ACLs says not to scan emails from localhost for spam.

So, the solution might be a hack, but it worked out perfectly. Starting fetchmail with the -S <servername> argument causes it to send emails to the given SMTP server rather than localhost. Using the real hostname of my server caused the "do not scan local mails" not to kick in and all mails received via fetchmail to be scanned again.

Problem fixed.

Moving my blog

2008-03-25T16:29:00.000+01:00

And yet another post today. As I am planning to take down my personal server in the next few weeks (maybe months) I have moved my blog to wordpress.com. A 301-redirect has been set up at http://sp.or.at/blog so people (and robots) are still able to find my blog.

Mails from Technorati not arriving: not obeying their own SPF rules

2008-03-25T15:59:00.000+01:00

As I was looking into problems with my mail server I noticed one more thing: I was wondering why I did not receive password recovery emails from Technorati. It seems as if they are not obeying their own SPF rules:


2008-03-25 14:46:23 H=nat-365m.technorati.com (t120.technorati.com) [208.66.64.4] F= rejected RCPT : Not authorized by SPF

Now I am wondering why someone sets up SPF for his mail domain when he is in fact sending emails from other IP addresses as well. Time to update your SPF rules Technorati...

Removing a lot of frozen mails from Exim’s mail queue

2008-03-25T15:40:00.000+01:00

After writing my last article, I started digging into my mail configuration and after doing a quick "mailq" noticed a lot of frozen messages in Exim's queue. After inspecting the logs and the mails themselves I noticed the problem was caused by a broken POP server I retrieve mails from periodically. A few days ago something went wrong on that server and all messages were marked as unread causing my fetchmail to re-fetch all of them (about 2.5K).

Now that my mail server is configured to do sender verification and a few very old mails came from domains or systems which are non-existent today about 50 mails ended up being frozen.

But how to remove all frozen mails from Exim's queue? I ended up using mailq | grep frozen to get a list of all messages (and more importantly their message IDs) and saved that to a file. I then wrote a minimalistic Python script attached to this article to delete all those messages. Consider the script a quick and dirty hack, but it might come in handy for some of you. Get it here.

More spam again?

2008-03-25T15:08:00.000+01:00

Right now I am asking myself if it just affects me or if more spam is sent out and less is detected by anti-spam software again.

I set up my mail server in February and noticed a decrease in spam mail delivered to my mailbox compared to my old system. However, in the past two weeks more and more spam mail has been delivered to my mailbox again. So is it just me, my system or the system's configuration or is everyone else receiving more spam again?

Anyways, it's about time to inspect the configuration of my mail system again...

Python IDEs tested

2008-03-19T12:26:00.000+01:00

In the past two days I have been playing around with various Python IDEs. It is not like I need a fully-fledged IDE, I'm fine with GNU Emacs to be honest. However, everyone is talking about IDE X and IDE Y and how they save so much time using these programs and how these programs assist them with hacking.

Well, I decided it was time to give a few IDEs a try. There were only two requirements I had: the IDE has to be Free Software and it has to run on GNU/Linux.

If you are planning to read on please be aware that this was no real test, but rather contains my observations regarding the IDEs I have tested, what I liked and did not like and if one surprised me enough to actually use it instead of my good old plain GNU Emacs.

Eclipse

As Java development in school is done with Eclipse and all teachers are more than happy with that program I gave it a try first. I head that there was some sort of Python IDE plugin and so I downloaded Eclipse 3.3. After a few problems keeping the bugger running for more than 5 minutes (seems like the default memory-usage configuration did not provide Eclipse with enough memory) I started downloading the PyDev plugin using the internal plugin download manager. This worked quite smoothly, however, it seemed a bit slow.
Now PyDev looks quite neat, but without the proprietary PyDev extensions it is rather useless and GNU Emacs gives me pretty much the same features.

CONCLUSION: Bloated, using a huge load of memory (Eight-Megabyte-And-Constantly-Swapping joke comes to my mind again), not offering a lot more features than GNU Emacs without proprietary PyDev Extensions.

OpenKomodo

After reading this post on lwn.net about OpenKomodo (note: the post says Komodo Edit, but that's proprietary software) and how it supports Python I gave it a try. I built it from Subversion trunk, which took some time. OpenKomodo is based on Mozilla and Gecko and if you ever built Firefox from source you should know that you can go and grab quite a few coffees while waiting for the build to finish.
The build system seems to be one specifically written for this application and so is a bit weird to use for people use to either GNU Autotools or Python's distutils. After the build process finished I was unable to find a way to install the application. The documentation only contains a note about using the build tool (black, "bk") with the "run" argument to start OpenKomodo.
At first everything looked quite nice. It supports Python quite well, including limited auto-completion support and so on and also supports, just like you would expect, tabbed-editing. After playing around in the source tree of one of my projects and trying to get used to "normal" keyboard shortcuts, such as Ctrl+s for saving a file, I had quite a few tabs open.
You probably know that having a lot of tabs open just leads to confusion and so I wanted to close all tabs but the currently active one and oops: that feature does not exist.
I then digged into the OpenKomodo source, added that feature, prepared a patch and tried to get it into the trunk: without luck as it seems. As noted in a comment to my bug report such features should go into extensions. As I am too lazy to write an extension just for this small patch and basic feature I am still trying getting the changes into trunk.
However, I abandoned OpenKomodo, as I found something better. First to my conclusion though:

CONCLUSION: Nice editor, but like Eclipse, quite bloated as it is based on Mozilla (memory leaks anyone?). Compile time is bad, again, because it is based on Mozilla. Getting simple patches into its trunk also seems to be a problem.

PIDA

I do not remember how or where I stumbled accross a reference to PIDA, but it sounded interesting. PIDA is a Python IDE, built using Python, with a lot of features.
Even though you cannot see this on the screenshots on the PIDA homepage it does not include its own editor. It rather makes use of an existing editor. It currently can embed either vim or GNU Emacs (you need CVS version 23.x or newer). As I was using GNU Emacs before this really caught my attention. I downloaded PIDA from mercurial and built it. Build time is less than 5 minutes on my machine, which is more than acceptable.
When starting PIDA for the first time it asks which editor you want to embed. I obviously chose Emacs there.
It seems like embedding Emacs is in an early stage right now. Even though everything seems to work PIDA embeds the whole GNU Emacs (GTK version) window, including the menu bar and the toolbar. This generates a weird look, as you have two menu- and toolbars, one belonging to PIDA and one belonging to GNU Emacs. No problem for me though, as it is rather a style-problem than a real one.
After opening up one of my projects I immediately noticed one thing: version control integration. I can confirm that Subversion is properly supported and works perfectly. Including reverting of files, updating the local copy, committing changes and viewing differences. This indeed is a great feature and I like it.
I played around a bit more and stumbled across the plugins. There are quite a few neat plugins, like a Trac integration plugin which allows you to view tickets inside the IDE or a TODO parser plugin, which parses comments containing "TODO:" or "XXX:" from files and gives you references to them.
Another useful plugin seems to be the Python Source Viewer, which displays all functions, classes and methods present in the current python file in a tree view.

CONCLUSION: The IDE I am most likely going to use for now. Why? Because it seems to be lightweight, uses GNU Emacs as embedded editor and comes with a proper feature set. I suggest everyone, even hardcore GNU Emacs users, to give PIDA a try. It looks worth it.

nwu development news #0

2008-01-16T22:49:00.000+01:00

So, today I am starting off with a new story series. The nwu development news.
Now what is this series about? Well, to make a long story short, it is about what has recently changed in nwu's codebase and how nwu is coming along.
Just a sidenote, the first story in this series is of course number 0, as real programmers start counting at 0. :-)

For those of you who are now wondering what nwu is or could be, I did write about nwu on this weblog already and the 'nwu - an introduction' post should give you a good idea of what it is.

So, what has changed recently? Basically I merged my changes back into trunk, which means that most of these things are going to be used now. This means that the application framework, the scheduler, the APT "Packages" file parser, support for gzip compression in both the SecureXMLRPC client and server and the brand-new RPC framework are either already being used, or are going to be used soon.

Except for the RPC framework, which would need to be adapted, and the application framework, which depends on nwu.common.config, all these pieces of code also work stand-alone and can be used in other python applications too.

Using parts of nwu in your project

2008-01-13T18:43:00.000+01:00

As I promised I am writing about nwu again. But instead of reporting on recent development efforts I would rather like to point something else out today: The nwu.common Python module contains code which can be used stand-alone in your applications. Some of the functions the module provides could come in handy, so I thought it was a good idea to let you know.

This article is going to explain the stand-alone nwu.common.* modules and their function.

nwu.common.SecureXMLRPC

I would like to start out introducing my "baby": SecureXMLRPC. As the name suggests it provides support for XML-RPC over https. You might think "but that's present in Python's SimpleXMLRPCServer and xmlrpclib already" and you are right about that. However, the Python implementation uses OpenSSL, whilst SecureXMLRPC provides both a server and client implementation using GnuTLS. Even though the OpenSSL version works perfectly fine you cannot link it to GPL-licensed code due to the OpenSSL license being incompatible to the GPL.
Furthermore SecureXMLRPC supports compression of the XML-RPC payload and as of today is aware of multi-threading.

nwu.common.config

The config module provides a single class, "Config". It basically is a slightly improved version of Python's SafeConfigParser class and allows passing a "default value" argument to its get() method which is returned in case the setting is not present in the config.

nwu.common.app

This is a simple application framework. It makes use of "nwu.common.config" for reading the config file. However, its main feature is a simple-to-use command-line parser, which parses not only arguments from the command-line (such as --do-something, -d or --configfile=filename), but also supports "commands" as they can be found in several applications, such as aptitude. This way it is easy to not only create top-level commands (like "aptitude search <package name>"), but also nested command structures (like "program computer get <computer name>", "program computer list", etc.).

nwu.common.certtool

The certtool module is a wrapper around GnuTLS' certtool application. It allows you to create private keys, certificate authorities, sign certificates, created certificate-signing-requests (CSRs) and sign such.
However, please note that it currently does not implement the full functionality certtool provides but rather only the pieces nwu needs.

nwu.common.aptmethod

nwu.common.aptmethod simplifies implementation of an APT transport/method in Python. It takes care of all communication done with apt itself and thus gives you the opportunity to implement a new method in a simple way.

nwu.common.apt

Finally there is also the apt module. This module currently contains code to parse APT Packages files.

nwu.common.scheduler

One of the latest additions to the common module. It provides a way of scheduling tasks for execution at a certain date/time and executing a task in a given interval.

Blue-GNU - News for Gnus

2008-01-13T12:02:00.000+01:00

I just stumbled accross what seems to be an interesting Free Software news site, named Blue-GNU, and wanted to let you know.

It also seems to be "GNU-approved" as there is a link to it on the (new) gnu.org frontpage.

nwu - an introduction

2008-01-13T11:57:00.000+01:00

This article should give you a brief overview of what network-wide updates, one of my projects, is about.

Network wide updates, or nwu, is a free software package licensed under the GPL (version 3 or later). It allows an administrator to remotely install software on and roll out security upgrades to managed computers. It is targeted at GNU/Linux systems using the Advanced Packaging Tool (APT) for package management and thus should run fine on all GNU/Linux distributions based on Debian GNU/Linux (such as gNewSense and all Ubuntu flavors).

It further gathers data about all managed computers in a database, which gives the administrator an overview about which systems are managed, the hardware configurations they have, the software installed on them and pending security updates.

The system is split into two major components: the server, which is the central management point, and the clients, usually referred to as agents, which report to the server.

Now let's go into nasty technical details. nwu is implemented in Python, using XML-RPC and python-gnutls (and this way the GnuTLS library) for secure communication between the server and its agents. It further makes use of X.509-certificate authentication and provides an integrated CA-system which simplifies use of this method a great deal.

Currently nwu is in an alpha development stage, but moving forward quite fast. We are expecting to be able to release a stable version quite soon.

The development team is made up of Yves Junqueira and myself right now, but we would be happy to accept contributions from others.

If you are interested in nwu development you should probably watch this blog, as I am going to write about recent developments here.