PROTOTYPING · TECHNOLOGY PATH

Connecting sensing, recognition and feedback

To test two-way communication, we built a hardware–software prototype combining Arduino-based sensing with computer-vision gesture recognition.

01

System architecture

Two linked loops support attention and interaction

HARDWARE LOOP

Sound detection & feedback

MicrophoneDetect nearby voice
ArduinoEvaluate threshold
Vibration alertWake OLED

When nearby speech exceeds the prototype threshold, Arduino triggers a private haptic cue and wakes the display.

VISION LOOP

Gesture recognition

CameraCapture hand pose
MediaPipeExtract landmarks
Classify gestureUpdate interface

The computer maps hand landmarks to a defined gesture and advances the visual conversation interface.

USB serial synchronizes computer-side recognition with Arduino-controlled feedback.
02

Initial circuit diagram

Early prototype wiring

Initial circuit diagram
The original circuit diagram is retained as documentation of the first integrated prototype.
03

Prototype modules

Core feedback components

Microphone module

Captures sound level for attention detection.

INPUT

Vibration motor

Provides a private tactile notification.

FEEDBACK

OLED display

Shows prompts, states and confirmations.

OUTPUT