Gesturism
A browser-based digital painting interface that leverage real-time body tracking as a creative tool
Try it out
Tools
Asana
HTML/CSS, JavaScript, React.js, Vercel, Mediapipe
Figma, Adobe Illustrator, Adobe Photoshop, Adobe Premiere Pro
Timeline
August 2024 - May 2025
Senior Thesis
Disciplines
User Interface Deisgn
User Experience
Software Engineering
Graphic Design
Challenge
Rooted in theories of embodied interaction, gestural communication, and computational media, this research critiques the constraints of conventional digital painting tools that limit creative expression to the hand and wrist
My Vision
Offer a novel systemthat integrates full-body kinesthetic data with customizable brush settings, thereby enabling artists, performers, and non-artists alike to draw with the entire body
Abstract
This thesis investigates how full-body movement can be materialized into digital brushstrokes through computer vision, proposing Gesturism, a browser-based digital painting interface that leverages real-time body tracking as a creative tool. The project synthesizes methodologies from human-computer interaction, creative coding, and movement studies. A user study (N=14) evaluated the system’s usability and expressiveness across diverse demographics, including dancers, athletes, and digital painters. The data reveals differentiated modes of interaction: some users approached the tool as a space for choreographic improvisation, while others prioritized visual precision. These findings affirm the value of movement not only as input but as material for dynamic, relational, and expressive design.
Beyond prototyping, this thesis contributes a conceptual and technical framework for designing with gesture as a medium, examining how movement data can be both structurally legible and aes- thetically open-ended. An accompanying exhibition and film documented how viewers engaged with Gesturism in situ, revealing social, cultural, and perceptual layers of par- ticipation. The project foregrounds co-participation, affective feedback, and embodied authorship, offering an alternative path forward for digital art systems grounded in the whole body.
Click to read full research paperArtist Collaboration
A One-Year Learning Journey
I went through the entire research process of validation of idea, methodologies, examination the body of work in computer vision to iterative development 0 -> 1. The duration of engineering the system took about 4 months, during which I continuously implementing 33 points of body tracking and improving accuracy and latency. I then went on to be a UI designer, applying my knowledge in Figma to design and refine the intuitiveness of navigating Gesturism. To validate my hypothesis, I conducted an user experience research (n=14) and summarize the findings in my research paper. I also presented this project at the senior art show, letting participants try Gesturism and showcasing collaboration work with another painter.
Current Stage
I didn’t stop after defending my thesis, I brought Gesturism to the Zingale Big Idea Entrepreneur Competition and won some seed funding. I’m currently refining Gesturism user interface and branding. Next step is to implement dynamic functions for artists and choreographers.
