AI-Based Sign Language to Speech Conversion System
ESP32 + AI Agent + IoT Dashboard + n8n Automation + Telegram Voice Alerts + Google Sheets + ThingSpeak
AI-Based Sign Language to Speech Conversion System
ESP32 + AI Agent + IoT Dashboard + n8n Automation + Telegram Voice Alerts + Google Sheets + ThingSpeak
1. Project Overview
Project Title
AI-Based Sign Language to Speech Conversion System using ESP32, Agentic AI, n8n Automation, Telegram Voice Alerts, Google Sheets, and ThingSpeak Cloud
Objective
The system recognizes sign language gestures using wearable sensors and converts them into:
Text
Speech
Telegram Voice Alerts
Cloud Data Logging
AI-based Analytics
The project combines:
IoT
Artificial Intelligence
Agentic Automation
Cloud Computing
Real-Time Monitoring
to assist speech-impaired individuals in communicating effectively.
2. System Architecture
Gesture Input
│
▼
Flex Sensors + MPU6050
│
▼
ESP32
│
▼
WiFi Network
│
┌────┼─────┐
▼ ▼ ▼
ThingSpeak
Google Sheets
n8n Workflow
│
▼
AI Agent
│
▼
Telegram Bot
│
▼
Voice Message Alert
3. Working Principle
Step 1
User performs a sign language gesture.
Example:
Gesture = HELP
Step 2
Flex sensors detect finger bending.
MPU6050 detects hand orientation.
Step 3
ESP32 reads sensor values.
Example:
Finger1 = 850
Finger2 = 300
Finger3 = 870
Finger4 = 900
Finger5 = 200
Step 4
ESP32 compares values with trained patterns.
if(F1>800 && F2<400)
{
gesture="HELP";
}
Step 5
Recognized text is sent to:
ThingSpeak
Google Sheets
n8n Webhook
Step 6
n8n AI Agent analyzes the message.
Example:
HELP
AI can generate:
Emergency Assistance Needed
Step 7
Telegram Bot sends:
Text Alert
User Requested Help
Voice Alert
Attention.
The user requires assistance immediately.
4. Hardware Components
Component Quantity
ESP32 Dev Board 1
Flex Sensors 5
MPU6050 Gyroscope 1
10kΩ Resistors 5
Breadboard 1
Jumper Wires Several
Power Bank 1
WiFi Router 1
Speaker (Optional) 1
OLED Display (Optional) 1
5. Component Description
ESP32
Main controller.
Functions:
Sensor Reading
WiFi Communication
Data Upload
Features:
Dual Core
240 MHz
WiFi
Bluetooth
Flex Sensors
Measure finger bending.
Range:
10kΩ - 40kΩ
Each finger has one sensor.
MPU6050
Measures:
Acceleration
Rotation
Hand Orientation
Used for gesture accuracy improvement.
6. Circuit Diagram
Flex Sensor Connections
Thumb → GPIO34
Index → GPIO35
Middle → GPIO32
Ring → GPIO33
Little → GPIO25
MPU6050 Connections
VCC → 3.3V
GND → GND
SDA → GPIO21
SCL → GPIO22
Complete Schematic
ESP32
+----------------+
Flex1 -------- GPIO34
Flex2 -------- GPIO35
Flex3 -------- GPIO32
Flex4 -------- GPIO33
Flex5 -------- GPIO25
MPU6050 SDA -- GPIO21
MPU6050 SCL -- GPIO22
VCC ---------- 3.3V
GND ---------- GND
+----------------+
7. Flowchart
START
|
Initialize ESP32
|
Connect WiFi
|
Read Sensors
|
Recognize Gesture
|
Upload Data
|
Trigger n8n Webhook
|
AI Agent Analysis
|
Telegram Alert
|
Store Data
|
Repeat
8. ESP32 Source Code
#include
#include
const char* ssid="YOUR_WIFI";
const char* password="PASSWORD";
String webhookURL=
"https://your-n8n-server/webhook/sign";
void setup()
{
Serial.begin(115200);
WiFi.begin(ssid,password);
while(WiFi.status()!=WL_CONNECTED)
{
delay(500);
}
Serial.println("Connected");
}
void loop()
{
int flex1=analogRead(34);
int flex2=analogRead(35);
int flex3=analogRead(32);
int flex4=analogRead(33);
int flex5=analogRead(25);
String gesture="Unknown";
if(flex1>3000 && flex2<1500)
{
gesture="HELP";
}
if(WiFi.status()==WL_CONNECTED)
{
HTTPClient http;
http.begin(webhookURL);
http.addHeader("Content-Type",
"application/json");
String payload=
"{\"gesture\":\""+gesture+"\"}";
http.POST(payload);
http.end();
}
delay(3000);
}
9. ThingSpeak Setup
Step 1
Create account at:
ThingSpeak
Step 2
Create New Channel
Fields:
Field1 = Gesture
Field2 = Confidence
Field3 = Power
Step 3
Copy Write API Key
Example:
ABC123XYZ
Step 4
ESP32 Upload
https://api.thingspeak.com/update?
api_key=ABC123XYZ
&field1=HELP
10. Google Sheets Integration
Method
ESP32 → n8n → Google Sheets
Create Sheet
Timestamp
Gesture
Confidence
Power
Example:
10:15 AM | HELP | 95% | 0.8W
11. Telegram Bot Setup
Step 1
Open Telegram
Search:
BotFather on Telegram
Step 2
Create Bot
/newbot
Step 3
Copy Token
123456:ABCDEF
Step 4
Get Chat ID
Send message:
/start
Store Chat ID.
12. Telegram Voice Notification
Example Voice Message
Attention.
The user has requested help.
Please assist immediately.
n8n Flow
Webhook
↓
AI Agent
↓
Text-to-Speech
↓
Telegram Send Voice
13. n8n Workflow Architecture
Webhook Trigger
│
▼
JSON Parse
│
▼
AI Agent
│
┌─────┴─────┐
▼ ▼
Sheets Voice
Update Alert
│
▼
ThingSpeak
14. Detailed n8n Nodes
Node 1
Webhook Trigger
POST /webhook/sign
Node 2
Set Node
{
"gesture":"HELP"
}
Node 3
AI Agent
Prompt:
Convert gesture into meaningful communication.
Node 4
Google Sheets
Append Row
Node 5
HTTP Request
Update ThingSpeak
Node 6
Telegram Send Message
User needs help
Node 7
Telegram Send Voice
Voice generated using TTS.
15. Sample n8n Workflow JSON
{
"nodes":[
{
"name":"Webhook",
"type":"n8n-nodes-base.webhook"
},
{
"name":"AI Agent",
"type":"@n8n/n8n-nodes-langchain.agent"
},
{
"name":"Google Sheets",
"type":"n8n-nodes-base.googleSheets"
},
{
"name":"Telegram",
"type":"n8n-nodes-base.telegram"
}
]
}
16. AI Power Consumption Prediction
Objective
Predict battery life.
Parameters:
WiFi Usage
CPU Usage
Sensor Usage
Transmission Count
Formula
P=VI
Where:
P = Power
V = Voltage
I = Current
Example
Voltage = 5V
Current = 0.18A
Power:
0.9W
Battery:
5000mAh
Runtime:
27 Hours
Approximate.
17. AI Agent Logic
Input
{
"gesture":"HELP"
}
AI Interpretation
Emergency communication request detected.
AI Output
{
"priority":"HIGH",
"message":"User needs help immediately"
}
18. Voice Automation Logic
Gesture
↓
AI Agent
↓
Generate Sentence
↓
Text-to-Speech
↓
Telegram Voice
Example:
HELP
becomes
Attention.
Emergency assistance requested.
19. Database Structure
Google Sheets Columns
Timestamp Gesture AI Message Priority Power
10:00 HELP Emergency HIGH 0.9W
20. Testing Procedure
Sensor Test
Check raw values.
Serial.println(flex1);
Gesture Test
Verify each sign.
HELP
YES
NO
WATER
FOOD
Cloud Test
Verify:
ThingSpeak Graphs
Google Sheet Entries
Telegram Alerts
21. Future Enhancements
AI Gesture Recognition
Replace rule-based system with:
CNN
LSTM
TinyML
for higher accuracy.
Multilingual Speech
Support:
English
Hindi
Telugu
Tamil
Mobile App
Features:
Live Speech
Dashboard
Analytics
Edge AI
Deploy model directly on ESP32 using:
TensorFlow Lite Micro
Edge Impulse
22. Deployment Guide
Step 1
Assemble glove.
Step 2
Upload ESP32 firmware.
Step 3
Configure WiFi.
Step 4
Create:
Telegram Bot
Google Sheet
ThingSpeak Channel
Step 5
Import n8n workflow.
Step 6
Start workflow.
Step 7
Wear glove and test gestures.
Expected Output Example
User Gesture
HELP
ESP32
Gesture Detected: HELP
Google Sheets
12:30 PM | HELP | HIGH
ThingSpeak
Gesture Frequency Graph
Power Consumption Graph
Telegram
Text:
🚨 User needs help immediately.
Voice:
Attention.
The user requires assistance.
AI Agent
Priority: HIGH
Suggested Action: Immediate Response
This architecture demonstrates a complete Industry 4.0/Agentic IoT solution integrating wearable sign-language recognition, ESP32 edge computing, AI interpretation, cloud analytics, workflow automation, voice notifications, and real-time monitoring.
Comments
Post a Comment