Whole-body interaction on a touch-sensitive floor
Oct 2010 — Jul 2011
GravitySpace is a new, camera-free approach to tracking objects and activities in smart rooms, using only touch-sensitive horizontal surfaces.
We built several large, high-precision multi-touch surfaces as prototypes. As a yearlong capstone project with Patrick Baudisch at Hasso Plattner Institute's chair for Human-Computer Interaction, we equipped all horizontal surfaces in a room with touch sensitivity, including chairs, tables, and the floor. As gravity pushes people and objects against these surfaces, it causes them to leave imprints, i.e., distinct pressure distribution across the surface. GravitySpace concludes from these imprints which objects or people occupy the room, how they are positioned, and which actions they are performing.
We implemented GravitySpace by building a 8m² FTIR glass floor (input and display resolution: 12 megapixels each), as well as several other pieces of active and passive touch-sensitive furniture. We then developed a framework on top of OpenCV in order to process imprint images and recognize objects and their spatial relationships above the surface.
We instrument all horizontal surfaces with touch sensors. From the resulting imprints (blue), we can determine the objects that caused them.
Our second-largest floor prototype. Its IR Cam and projector are mounted in the foreground. Imprint images are projected back onto the floor with a slight offset to make them visible from above.
We can identify people, objects, and even pressure distribution from imprints on the surfaces, creating a '3D model' of the scene above the surfaces.
FTIR lets us to visualize pressure: Light is sent lengthwise through a glass pane, and is deflected out of the pane only where an object touches the surface.
Pressure-propagating furniture, such as this cube seat, allow the floor to register pressure distribution indirectly. This seat can be used as a generic 'weight shift' input device, e.g. to control a racing game.
- Prof. Patrick Baudisch
- Paul Meinhardt
- Jossekin Beilharz
- Nicholas Wittstruck
- Marcel Kinzel
- Franziska Boob
- Jan Burhenne
- "Wenn die Fußmatte die Tür öffnet", Berliner Zeitung. February, 19, 2011
- "Multitoe" - RBB (Nationwide Public TV), 4min, RBB Zibb. January, 5, 2011