PENCAK
PENCAK — a living archive of Silat Seni Gayong
About
A living archive of Silat Seni Gayong: a full-body VR trainer built in Unity for Meta Quest 3. Existing VR martial arts apps track only arms and hands, which breaks down for Silat, where evasion is entirely footwork — so PENCAK pairs the headset with HTC Vive trackers to read stance, stepping and foot position, and validates placement live on a grid mat.
What I Did
Leading the programming for Team IMMERSA: extending Quest 3 with Vive trackers for below-the-waist coverage, building the grid-mat system that validates foot placement in real time, and integrating Convai to turn onboarding into a conversation with a virtual Guru instead of a setup form.
Highlights
- Lower-body tracking via HTC Vive trackers on Meta Quest 3
- Grid mat that validates stance and foot placement in real time
- Next step shown as a yellow footprint, turning green when landed
- Conversational AI Guru built on Convai for voice onboarding
- Guru answers questions and repeats demonstrations mid-session
- Spoken name, age and height drive body calibration and difficulty
- Three modules: Asas Serangan, Asas Elakkan, Elak & Serang
- Scoring across hand movement, footwork, stance, timing and accuracy
Technical Details
Silat Seni Gayong lives in muscle memory — its forms were never meaningfully written down, and correction only happens when a teacher is watching from the right angle at the right moment, so a student can drill a wrong stance a thousand times and never know. Benchmarking the closest VR reference points exposed two fatal gaps for this discipline: KungFu VR teaches through movement but tracks the upper body only, and FitXR's virtual instructor demonstrates without ever being able to answer a question. Both are disqualifying here, because evasion is footwork and correction has to be conversational. The build answers each directly: Vive trackers extend Quest 3 below the waist to supply real stance and stepping data, feeding a grid mat that marks the player's position, projects the next step as a yellow footprint, and turns it green once landed — so a wrong placement flags immediately instead of being silently reinforced. Convai powers a virtual Guru who gathers the player's details by voice, then stays available through the session. The two halves have to agree: the Guru's correction is only credible if the tracking underneath is genuinely reading the stance right, so calibration and feedback logic can't be developed independently — a false positive on foot placement turns the Guru into a liar, which is worse than giving no feedback at all.
More projects
