TactileSight

Beyond your eyes: a haptic headband that lets you feel the environment a cane can't reach.

See it in action

_{Live depth feed → 21-cell haptic grid → real-time motor output, all on one dashboard.}

The problem → The solution

A white cane reaches about a metre and tells you only that something is there, not that the doorway is on your left, or that the thing ahead is a person and not a pillar. 🦯 AI photo assistants can describe a scene, but they guess at distance from a flat image, and a confident wrong guess is worse than no answer at all when you're walking.

TactileSight splits the problem in two: a depth camera measures the world in millimetres, never guesses, and translates it straight into vibration on your skin. That is the reflex layer. A phone app adds the what, names, signs and spoken answers, on demand and in your own language. 🧭

How it works

 Orbbec Astra Pro   ──depth──▶   21/27-cell grid   ──MsgPack-RPC──▶   STM32U585   ──I2C──▶   PCA9685 ×2   ──▶   27 motors
  depth camera         (QRB2210,      reduction         Arduino          (drives            (2 boards)      (buzz = near)
                        Hexagon NPU)                     Router Bridge)   the motors)
                            │
                            ▼
                    Qwen3-VL-4B + YOLOv11
                    (on-device, NPU)
                            │
                            ▼
                    Sarvam ASR / TTS
                    (11 Indian languages)
                            │
                            ▼
                     spoken answer

Sense. The Astra Pro captures colour + depth together, one frame at a time.
Feel. Depth is reduced to a haptic grid and pushed straight to vibration motors, no phone required.
Understand. On demand, a tap or hold on the phone runs the frame through an on-device vision model and YOLOv11 for what's ahead, and the depth sensor for how far.
Speak. The answer is translated and spoken back in the user's own language.

Prototype gallery

_{27-motor haptic pad, wired}	_{Motor grid sewn into the headband}
_{Soldering the motor wiring}	_{Depth camera mounted, demo-ready}
_{Live dashboard: depth feed + haptic grid}

Tech stack

Arduino UNO Q (QRB2210 arm64 Linux) STM32U585 Cortex-M33 Hexagon NPU / QAIRT Qwen3-VL-4B YOLOv11 (TFLite) Orbbec Astra Pro Plus Sarvam AI (ASR/TTS/Translate) Kotlin · CameraX (Android) Python (server) PCA9685 PWM drivers MsgPack-RPC / Arduino Router Bridge NFC

Key features

Measures real obstacle distance from depth data. Never guesses, never invents a number
Buzzes a 27-motor grid on the band in real time, no phone needed for the reflex layer
One button on the phone: tap to describe what's ahead, hold to ask a question
Reads signs and text aloud, so a corridor becomes navigable instead of just described
Speaks and understands 11 Indian languages, confirmed back in the language just chosen

Architecture

flowchart LR
    A[Orbbec Astra Pro<br/>depth + colour] -->|USB host| B[QRB2210 / Linux]
    B -->|21-cell grid| C[MsgPack-RPC<br/>Router Bridge]
    C --> D[STM32U585]
    D --> E[PCA9685 x2]
    E --> F[27 vibration motors]
    D --> G[LED matrix mirror]
    B -->|on tap/hold| H[Qwen3-VL-4B<br/>Hexagon NPU]
    B --> I[YOLOv11<br/>object + distance]
    H --> J[Sarvam ASR/TTS<br/>11 languages]
    I --> J
    J --> K[Spoken answer]

Quick start

Phone app

cd android
JAVA_HOME=/path/to/android-studio/jbr ./gradlew assembleDebug

Needs JDK 17 to 21, and a sarvam.api.key in android/local.properties. The QAIRT model bundle is pushed over adb, not bundled in the APK:

adb push <bundle>/. /sdcard/Android/data/com.tactilesight/files/models/geniex/

Board / haptic server

ssh arduino@<board-ip>            # password: vidhu123
sudo /usr/local/bin/usb-role host # camera needs host mode
systemctl status haptic-demo
# dashboard: http://<board-ip>:8081

First-time board setup: linux/setup.sh.

Team

Built by the TactileSight Team for the Qualcomm Snapdragon Multiverse Hackathon, Noida, 18-19 July 2026.

$ TactileSight