PintAR / PrototipAR: What You Sketch Is What You Get
A mixed fidelity prototyping and authoring tool that exploits situated sketching in mixed reality — combining digital pen input with head-mounted displays so designers can author AR experiences without code.
The Problem
Building AR experiences has traditionally required deep programming knowledge. Toolkits like the Microsoft Mixed Reality Toolkit, Unity3D, and Unreal Engine are powerful but create steep barriers for designers, UX researchers, and non-programmers. This gap slows down iteration cycles and means the people who best understand the design problem — the designers — often cannot directly express their ideas in the medium they are designing for.
I wanted to change that. What if designers could sketch an AR experience the same way they sketch on paper — quickly, directly, and without writing a single line of code?
Our Solution
PintAR (and its evolution, PrototipAR) is a mixed fidelity prototyping and authoring tool that exploits situated sketching in mixed reality. It brings designers' ideas to life by combining a digital pen for sketching on a tablet with a head-mounted display for visualizing and interacting with the sketched content directly in the physical environment. The core philosophy is simple: what you sketch is what you get.
How It Works
- Sketch on tablet: Designers draw 2D content using a digital pen interface, creating visual elements and UI components
- See in AR: Sketched elements are immediately rendered in the head-mounted display, placed in the physical environment
- Define behaviors: Interactive behaviors can be specified without coding through a state machine system that manages experience stages
- Debug spatially: An aerial overview feature helps designers debug and refine the spatial layout of their AR prototypes
Technology Stack
- Node.js backend for real-time synchronization
- Unity3D with C# for the AR rendering pipeline
- Fabric.js for the 2D sketching canvas
- WebSocket communication between tablet and HMD
My Contribution
I led the design and development of the PintAR/PrototipAR system, from the initial concept of combining pen-based sketching with AR head-mounted displays through to the implementation of the full prototyping pipeline. I built the real-time synchronization layer, co-designed the interaction model with the team, and led the demonstration and evaluation of the system at CHI 2019 and DIS 2019.
Publications
Gasques, D., Johnson, J., Sharkey, T., & Weibel, N. (2019). What you sketch is what you get: Quick and easy augmented reality prototyping with PintAR. CHI 2019 Interactivity.
Gasques, D., Johnson, J., Sharkey, T., & Weibel, N. (2019). PintAR: Sketching spatial experiences in augmented reality. DIS 2019 Demo.
Demos
DIS 2019 Demo
CHI 2019 Interactivity Demo
Team
Danilo Gasques, Ph.D., Tommy Sharkey, Janet Johnson, Nadir Weibel, Ph.D.