2012-02-24

ptrace-based security just does not work

In 2009 I wrote about building a ptrace-based sandboxing system named "ujail", including some basic proof of concepts.

I have been thinking about this idea for a long time now, but sadly did not have the time to implement it - until now.
Right now I am working on this idea again and whilst doing some research I came across a thread on the linux-kernel mailing list.
At first a problem with 64-bit binaries trapping into 32-bit syscall handling code via int 80 got me there. While this is awkward and keeps one from implementing a sandbox in userspace (due to not being able to access TS_COMPAT, as described in the thread) it led me to something else - a more severe problem.
Unfortunately I cannot remember who wrote this and am unable to recover the actual mail (if someone finds it I would be happy if you notified me), but someone mentioned race conditions when using ptrace as a security measure.

In short I came up with a proof of concept which works around possible limitations imposed by a ptrace-based security mechanism. For those in a hurry: you can find the code of the proof of concept at github.
In the following parts of this article I would like to elaborate on the problem and how the proof of concept code exploits it.