The design below uses an ESP32-CAM to capture hand gestures, an AI vision model to recognize a predefined gesture vocabulary, n8n as the automation/agent layer, text-to-speech for voice generation, Telegram for remote voice alerts, Google Sheets for logging, ThingSpeak for IoT visualization, and a web dashboard for live monitoring.
1. Project Title
AI-Powered Hand Gesture to Speech Converter and Agentic IoT Communication System
Short title
AI Gesture-to-Speech ESP32 IoT with n8n Automation, Telegram Voice Alerts, Google Sheets and ThingSpeak Dashboard
2. Abstract
People who cannot communicate through normal speech may use sign language, gestures, writing, or communication devices. This project proposes an IoT-based assistive communication system capable of recognizing predefined hand gestures and converting them into meaningful text and synthesized speech.
An ESP32-CAM captures an image of the user's hand gesture. The image is transmitted through Wi-Fi to an n8n Webhook. n8n acts as the automation and agentic layer. An AI vision model analyzes the image and identifies the gesture, for example:
- HELLO
- YES
- NO
- HELP
- WATER
- FOOD
- MEDICINE
- EMERGENCY
- THANK YOU
- I NEED HELP
The recognized gesture is converted into text. The system can then:
- Display the text on a web dashboard.
- Generate speech using a text-to-speech service.
- Send the generated voice message through Telegram.
- Record the event in Google Sheets.
- Send numerical event information to ThingSpeak.
- Trigger special actions for emergency gestures.
- Allow an AI agent to decide what automation should occur.
The ESP32 can also provide a local webpage showing the latest recognized gesture, system status, Wi-Fi status, confidence, and communication history.
ESP32 supports Wi-Fi station mode for Internet-connected IoT applications and can make HTTP/S requests, making it suitable as the edge device in this architecture.
3. Important Design Decision
There are two possible architectures.
Architecture A — Recommended
ESP32-CAM
|
| Image
v
n8n Webhook
|
v
AI Vision
|
v
Gesture JSON
|
+------> Text
|
+------> AI Agent
| |
| +--> Telegram
| +--> Google Sheets
| +--> ThingSpeak
| +--> TTS
|
v
ESP32 / Web Dashboard
Architecture B — Edge AI
Camera
|
ESP32
|
TinyML / TensorFlow Lite
|
Gesture
|
n8n
Architecture B is more difficult because the ESP32 has limited memory and computing resources.
For a final-year project/demo, Architecture A is considerably easier to implement and demonstrate.
4. System Objectives
The project has the following objectives:
Primary objective
Convert predefined hand gestures into speech automatically.
Secondary objectives
- Real-time gesture recognition.
- IoT connectivity using ESP32.
- AI-based image interpretation.
- Automated workflows using n8n.
- Voice notification through Telegram.
- Cloud data logging.
- Google Sheets database.
- ThingSpeak visualization.
- Web-based monitoring.
- Emergency alert mechanism.
- AI-agent-based decision making.
5. Proposed Gesture Vocabulary
Start with a small vocabulary rather than attempting complete sign-language translation.
| Gesture | Meaning | Action |
|---|---|---|
| OPEN PALM | HELLO | Normal message |
| THUMBS UP | YES | Normal message |
| THUMBS DOWN | NO | Normal message |
| FIST | STOP | Normal message |
| POINT UP | I NEED HELP | Alert |
| HEART | THANK YOU | Normal |
| WATER gesture | I NEED WATER | Normal |
| FOOD gesture | I NEED FOOD | Normal |
| MEDICAL gesture | MEDICAL HELP | High priority |
| EMERGENCY gesture | EMERGENCY | Critical alert |
Why start with fixed gestures?
A complete sign language is not simply a collection of static hand poses. It can involve:
- hand shape
- movement
- position
- orientation
- facial expressions
- two-handed movements
- timing
- context
Therefore, this project should initially be described as a predefined gesture recognition system, not a universal sign-language translator.
6. Overall Block Diagram
┌─────────────────────┐
│ USER / HAND │
│ GESTURE │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ ESP32-CAM │
│ │
│ Camera + Wi-Fi │
└──────────┬──────────┘
│
HTTP/HTTPS
│
▼
┌─────────────────────┐
│ n8n WEBHOOK │
│ │
│ Workflow Trigger │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ AI VISION │
│ Gesture Recognition │
└──────────┬──────────┘
│
JSON Result
│
▼
┌─────────────────────┐
│ AI AGENT │
│ Decision / Routing │
└──────────┬──────────┘
│
┌──────────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
┌────────────┐ ┌─────────────┐ ┌─────────────┐
│ Telegram │ │ Google │ │ ThingSpeak │
│ Voice │ │ Sheets │ │ Dashboard │
└────────────┘ └─────────────┘ └─────────────┘
│ │ │
└──────────────────┼──────────────────┘
│
▼
┌─────────────────────┐
│ WEB DASHBOARD │
│ │
│ Gesture │
│ Text │
│ Confidence │
│ Status │
│ History │
└─────────────────────┘
7. Detailed Working Principle
Step 1 — User performs a gesture
The user places their hand in front of the ESP32-CAM.
For example:
✋
|
|
"HELP"
The camera captures the hand.
8. Step 2 — ESP32-CAM captures image
The ESP32-CAM captures a JPEG image.
The ESP32 uses Wi-Fi to connect to the Internet and sends the image to the n8n webhook.
ESP32 supports Wi-Fi station mode, which is the appropriate mode when the board needs Internet connectivity.
9. Step 3 — n8n receives the image
n8n Webhook:
POST /gesture
Example:
https://YOUR-N8N-DOMAIN/webhook/gesture
The incoming request contains:
device_id
timestamp
image
10. Step 4 — AI Vision Recognition
The AI receives the image and is instructed to return structured information.
Example:
{
"gesture": "HELP",
"text": "I need help",
"confidence": 0.94,
"priority": "HIGH"
}
Modern vision-capable AI APIs can accept image inputs for classification and visual interpretation.
11. AI Prompt
Use a strict prompt such as:
You are a hand gesture recognition system for an assistive
communication device.
Recognize ONLY gestures from this vocabulary:
HELLO
YES
NO
STOP
HELP
WATER
FOOD
MEDICINE
THANK_YOU
EMERGENCY
UNKNOWN
Return ONLY valid JSON.
Required format:
{
"gesture": "HELP",
"text": "I need help",
"confidence": 0.94,
"priority": "HIGH"
}
Rules:
1. Do not invent gestures.
2. If the image is unclear, return UNKNOWN.
3. confidence must be between 0 and 1.
4. EMERGENCY must have priority CRITICAL.
5. HELP must have priority HIGH.
6. Do not infer sensitive information about the person.
12. Step 5 — AI Agent
The n8n AI Agent becomes the decision-making layer.
For example:
Gesture
|
v
┌─────────────┐
│ AI AGENT │
└──────┬──────┘
|
┌─────────┼─────────┐
| | |
v v v
NORMAL HIGH CRITICAL
| | |
v v v
Log Log + Log +
Sheet Telegram Telegram
| Voice
v
TTS
The agent can have tools such as:
Tool 1: log_to_google_sheets
Tool 2: update_thingspeak
Tool 3: send_telegram_text
Tool 4: send_telegram_voice
Tool 5: update_dashboard
Tool 6: notify_emergency_contact
13. Agent Decision Example
Input:
{
"gesture": "EMERGENCY",
"text": "I need emergency assistance",
"confidence": 0.97,
"priority": "CRITICAL"
}
Agent decision:
1. Save event.
2. Generate speech.
3. Send Telegram voice alert.
4. Send Telegram text.
5. Update ThingSpeak.
6. Mark event CRITICAL.
7. Update dashboard.
14. n8n Workflow
The complete n8n workflow should look like:
┌───────────────┐
│ Webhook │
│ /gesture │
└───────┬───────┘
│
▼
┌───────────────┐
│ Extract Image │
│ + Device ID │
└───────┬───────┘
│
▼
┌────────────────┐
│ AI Vision │
│ Recognition │
└───────┬────────┘
│
▼
┌────────────────┐
│ JSON Parser │
└───────┬────────┘
│
▼
┌────────────────┐
│ AI Agent │
└───────┬────────┘
│
├───────────────┐
│ │
▼ ▼
Google Sheets ThingSpeak
│ │
└───────┬───────┘
│
▼
Priority Check
│
┌──────┴───────┐
│ │
NORMAL EMERGENCY
│ │
▼ ▼
TTS TTS + Alert
│ │
└──────┬───────┘
▼
Telegram
│
▼
Web Dashboard
n8n has a built-in Telegram integration for automating Telegram operations.
15. Hardware Required
Main components
| Component | Quantity |
|---|---|
| ESP32-CAM AI Thinker | 1 |
| FTDI USB-to-TTL programmer | 1 |
| 5V power supply | 1 |
| LED | 1 |
| 220Ω resistor | 1 |
| Push button | 1 |
| Buzzer | 1 |
| Breadboard | 1 |
| Jumper wires | Several |
| Wi-Fi router/hotspot | 1 |
Optional
- OLED display
- Speaker
- MAX98357A I2S amplifier
- MicroSD card
- PIR sensor
- emergency button
- rechargeable battery
16. Hardware Schematic
A simple version:
┌──────────────────┐
│ ESP32-CAM │
│ │
│ Camera │
│ │
│ GPIO │
└───────┬──────────┘
│
┌─────────┼──────────┐
│ │ │
▼ ▼ ▼
LED BUZZER BUTTON
220Ω
│
GND
ESP32-CAM
│
│ Wi-Fi
▼
Internet
│
▼
n8n
17. Optional Speaker Circuit
For local speech output:
ESP32
|
I2S
|
v
MAX98357A
|
v
Speaker
However, for the first version I recommend:
Gesture
↓
n8n
↓
TTS
↓
Telegram Voice
This avoids implementing audio decoding and I2S playback on the ESP32 initially.
18. Telegram Architecture
AI Agent
|
v
Text
|
v
TTS
|
v
MP3/voice
|
v
Telegram Bot
|
v
Caregiver / Family Member
Telegram's Bot API provides sendVoice for voice messages. Current Bot API documentation supports voice messages in formats including OGG/Opus, MP3 and M4A, subject to the API's file handling requirements.
19. Telegram Example
Suppose the user makes:
HELP gesture
Telegram receives:
🚨 ASSISTIVE DEVICE ALERT
Gesture: HELP
Message:
"I need help."
Confidence:
94%
Priority:
HIGH
And below it:
🔊 Voice message
20. Text-to-Speech
A TTS API converts:
"I need help."
into:
speech.mp3
For example, OpenAI's current audio API provides:
POST /v1/audio/speech
for generating speech from text. It supports multiple audio formats including MP3 and Opus.
Example request:
{
"model": "gpt-4o-mini-tts",
"voice": "alloy",
"input": "I need help.",
"response_format": "mp3"
}
21. Google Sheets Database
Create a spreadsheet called:
AI Gesture Communication Log
Columns:
| Timestamp | Device ID | Gesture | Text | Confidence | Priority | Status |
|---|---|---|---|---|---|---|
| 18-08-2026 21:20 | ESP32CAM01 | HELP | I need help | 0.94 | HIGH | Sent |
| 18-08-2026 21:22 | ESP32CAM01 | WATER | I need water | 0.91 | NORMAL | Sent |
| 18-08-2026 21:25 | ESP32CAM01 | EMERGENCY | Emergency | 0.98 | CRITICAL | Alerted |
This gives you a complete historical database.
22. ThingSpeak Configuration
Create a ThingSpeak channel.
Suggested fields:
Field 1 = Gesture ID
Field 2 = Confidence
Field 3 = Priority
Field 4 = Device Status
Field 5 = Event Count
ThingSpeak's REST API allows channel data to be updated using GET or POST requests through https://api.thingspeak.com/update.
Example:
https://api.thingspeak.com/update.json
POST:
api_key=YOUR_WRITE_API_KEY
field1=5
field2=0.94
field3=2
field4=1
field5=27
23. Gesture ID Table
Use numerical IDs for ThingSpeak.
1 = HELLO
2 = YES
3 = NO
4 = STOP
5 = HELP
6 = WATER
7 = FOOD
8 = MEDICINE
9 = THANK_YOU
10 = EMERGENCY
0 = UNKNOWN
24. ThingSpeak Flow
Gesture Recognition
|
v
JSON
|
v
n8n
|
v
HTTP Request
|
v
ThingSpeak API
|
v
Channel
|
v
Charts
25. ESP32-CAM Firmware
