🤖 AI-Based Emotion Recognition Robot for Human Interaction
ESP32 + AI Agent + IoT + n8n Automation + Telegram Voice Alerts + Google Sheets + ThingSpeak Cloud Dashboard
Complete Project Description, Components, Circuit Diagram, Flowchart, ESP32 Source Code, n8n Workflow, Telegram Integration, Google Sheets, ThingSpeak, AI Power Prediction and Deployment Guide
1. Project Overview
The AI-Based Emotion Recognition Robot for Human Interaction is an intelligent robotic system designed to recognize human emotional states and respond appropriately through voice, movement, display and IoT automation.
The system combines artificial intelligence, embedded systems, robotics, Internet of Things technology, cloud monitoring and workflow automation.
Complete Technology Stack:
- ESP32 IoT Controller
- AI Emotion Recognition
- AI Agent Decision-Making
- n8n Workflow Automation
- Telegram Voice Notifications
- Google Sheets Data Logging
- ThingSpeak Cloud Dashboard
- PHP IoT Web Dashboard
- Power Consumption Monitoring
- AI Power Consumption Prediction
2. Main Project Objective
The main objective of this project is to develop an intelligent human-interaction robot capable of recognizing human emotions and responding automatically.
The system detects emotions such as:
- 😊 Happy
- 😢 Sad
- 😡 Angry
- 😐 Neutral
- 😨 Fear
- 😲 Surprise
- 😴 Tired or Sleepy
After detecting the emotional state, the robot analyzes the information using an AI Agent and performs a suitable action.
3. Complete System Architecture
+-------------------------+
| HUMAN USER |
| Face / Voice / Sensors |
+------------+------------+
|
v
+-------------------------+
| EMOTION RECOGNITION |
| AI Model / Camera / |
| Voice Analysis |
+------------+------------+
|
v
+-------------------------+
| ESP32 CONTROLLER |
| |
| Emotion Data |
| Sensor Data |
| Power Monitoring |
+------------+------------+
|
| Wi-Fi
v
+-------------------------+
| AI AGENT |
| Emotion Interpretation |
| Decision Making |
+------------+------------+
|
v
+-------------------------+
| n8n AUTOMATION |
+-----+-----+-----+-------+
| | |
v v v
Telegram Google ThingSpeak
Voice Sheets Cloud
Alert Logs Dashboard
|
v
+-------------------------+
| WEB DASHBOARD |
| Emotion Status |
| Robot Status |
| Power Usage |
| AI Prediction |
+-------------------------+
4. Hardware Components
| Component | Function |
|---|---|
| ESP32 Development Board | Main IoT controller |
| ESP32-CAM or Camera Module | Facial emotion recognition |
| OLED Display | Displays detected emotion |
| Microphone Module | Voice and audio input |
| MAX98357A / DFPlayer Mini | Audio output |
| Speaker | Voice response |
| Servo Motors | Robot head or hand movement |
| Motor Driver | Controls DC motors |
| DC Motors | Robot movement |
| Ultrasonic Sensor | Obstacle detection |
| DHT22 / BME280 | Temperature and humidity monitoring |
| INA219 | Voltage and current measurement |
| Battery | Power supply |
| LEDs | Emotion indication |
| Buzzer | Alert indication |
5. ESP32 Pin Configuration
ESP32 GPIO 21 -> I2C SDA ESP32 GPIO 22 -> I2C SCL ESP32 GPIO 34 -> Analog Sensor Input ESP32 GPIO 25 -> Servo Motor ESP32 GPIO 26 -> Motor Driver IN1 ESP32 GPIO 27 -> Motor Driver IN2 ESP32 GPIO 32 -> Ultrasonic TRIG ESP32 GPIO 33 -> Ultrasonic ECHO ESP32 GPIO 4 -> Emotion LED ESP32 GPIO 16 -> Audio RX ESP32 GPIO 17 -> Audio TX
6. Circuit Schematic Diagram
+--------------------+
| ESP32 |
| |
SDA ------| GPIO 21 |
SCL ------| GPIO 22 |
| |
Emotion LED ----------| GPIO 4 |
| |
Servo Signal ---------| GPIO 25 |
| |
Motor IN1 ------------| GPIO 26 |
Motor IN2 ------------| GPIO 27 |
| |
Ultrasonic TRIG ------| GPIO 32 |
Ultrasonic ECHO ------| GPIO 33 |
| |
INA219 SDA -----------| GPIO 21 |
INA219 SCL -----------| GPIO 22 |
+---------+----------+
|
+------------------+----------------+
| | |
v v v
+--------------+ +-------------+ +-------------+
| OLED Display | | INA219 | | BME280 |
| | | Power | | Environment |
+--------------+ | Sensor | +-------------+
+-------------+
+------------------+
| Camera / AI |
| Emotion Module |
+---------+--------+
|
v
+-------------+
| Emotion |
| Detection |
+-------------+
ESP32 -------- Wi-Fi -------- Internet
|
+------------------------+--------------------+
| | |
v v v
n8n Server ThingSpeak Web Server
|
+--------+----------+---------+
| | |
v v v
Telegram Google AI Agent
Voice Alert Sheets
7. Complete Working Principle
Step 1: Human Interaction
The user interacts with the robot by looking at the robot, speaking to the robot, showing facial expressions or asking questions.
Step 2: Emotion Detection
The camera or AI emotion-recognition system analyzes the facial expression and identifies the emotional state.
{
"emotion": "happy",
"confidence": 0.94
}
Step 3: ESP32 Processing
The ESP32 receives the emotion data and combines it with sensor information.
Emotion = SAD Confidence = 89% Temperature = 29.5 °C Battery = 7.4 V Current = 320 mA
Step 4: AI Agent Decision
The AI Agent analyzes the emotion and generates an appropriate response.
Input: Emotion = Sad Confidence = 89% AI Output: Response = "You seem a little sad. I am here with you." Action = Comfort Mode Robot Movement = Slow Head Movement Alert Level = Medium
Step 5: n8n Automation
The ESP32 sends the information to the n8n webhook. n8n processes the data and automatically performs multiple actions.
- Receives ESP32 data
- Processes JSON
- Sends data to AI Agent
- Generates intelligent response
- Sends Telegram notification
- Generates voice notification
- Stores data in Google Sheets
- Uploads data to ThingSpeak
- Updates the IoT web dashboard
8. Complete System Flowchart
+-------------+
| START |
+------+------+
|
v
+-------------+
| Initialize |
| ESP32 |
+------+------+
|
v
+-------------+
| Connect WiFi|
+------+------+
|
v
+-------------+
| Capture Face|
| Voice Data |
+------+------+
|
v
+-------------+
| Detect |
| Emotion |
+------+------+
|
v
+---------------------+
| Confidence > 60% ? |
+----------+----------+
|
+--------+--------+
| |
NO YES
| |
v v
+---------------+ +--------------+
| Retry Capture | | Send ESP32 |
+---------------+ | Emotion Data |
+------+-------+
|
v
+--------------+
| n8n Webhook |
+------+-------+
|
v
+--------------+
| AI Agent |
| Decision |
+------+-------+
|
+----------------+----------------+
| | |
v v v
Telegram Google Sheets ThingSpeak
Voice Alert Data Logging Cloud
| | |
+----------------+---------------+
|
v
+--------------+
| Robot |
| Response |
+------+-------+
|
v
+--------------+
| Continue Loop|
+--------------+
9. Emotion Recognition Logic
The emotion-recognition system may use one of the following architectures.
Option A: Cloud AI
Camera | v ESP32-CAM | v AI API | v Emotion Result | v ESP32
Option B: Edge AI
Camera | v ESP32-S3 / Edge AI Model | v Emotion Result
Option C: Computer Vision Computer
Camera | v Python AI Computer | v Emotion Detection | v ESP32
10. Emotion Response Table
| Emotion | AI Response | Robot Action | Alert Level |
|---|---|---|---|
| Happy | Encouraging message | Fast movement | Low |
| Sad | Supportive message | Slow movement | Medium |
| Angry | Calm response | Stop movement | High |
| Fear | Reassurance | Approach slowly | High |
| Surprise | Ask what happened | Head movement | Low |
| Neutral | Greeting | Normal operation | Low |
11. ESP32 Source Code
#include <WiFi.h>
#include <HTTPClient.h>
#include <ArduinoJson.h>
const char* WIFI_SSID = "YOUR_WIFI_NAME";
const char* WIFI_PASSWORD =
"YOUR_WIFI_PASSWORD";
const char* N8N_WEBHOOK =
"https://YOUR_N8N_SERVER/webhook/emotion-robot";
#define EMOTION_LED 4;
float voltage = 7.4;
float current = 0.32;
float power = 0.0;
void setup() {
Serial.begin(115200);
pinMode(
EMOTION_LED,
OUTPUT
);
WiFi.begin(
WIFI_SSID,
WIFI_PASSWORD
);
Serial.print(
"Connecting to WiFi"
);
while (
WiFi.status()
!= WL_CONNECTED
) {
delay(500);
Serial.print(".");
}
Serial.println();
Serial.println(
"WiFi Connected"
);
Serial.println(
WiFi.localIP()
);
}
void loop() {
String emotion =
detectEmotion();
float confidence =
0.92;
power =
voltage * current;
sendEmotionData(
emotion,
confidence,
voltage,
current,
power
);
delay(30000);
}
String detectEmotion() {
String emotions[] = {
"happy",
"sad",
"neutral",
"angry",
"surprise"
};
int index =
random(0, 5);
return emotions[index];
}
void sendEmotionData(
String emotion,
float confidence,
float voltage,
float current,
float power
) {
if (
WiFi.status()
== WL_CONNECTED
) {
HTTPClient http;
http.begin(
N8N_WEBHOOK
);
http.addHeader(
"Content-Type",
"application/json"
);
DynamicJsonDocument doc(
1024
);
doc["device_id"] =
"EMOTION_ROBOT_001";
doc["emotion"] =
emotion;
doc["confidence"] =
confidence;
doc["voltage"] =
voltage;
doc["current"] =
current;
doc["power"] =
power;
doc["robot_status"] =
"ONLINE";
String jsonData;
serializeJson(
doc,
jsonData
);
int responseCode =
http.POST(
jsonData
);
Serial.println(
"HTTP Response:"
);
Serial.println(
responseCode
);
http.end();
}
}
12. ESP32 JSON Data Format
{
"device_id":
"EMOTION_ROBOT_001",
"emotion":
"sad",
"confidence":
0.89,
"temperature":
29.5,
"humidity":
58,
"battery_voltage":
7.42,
"current":
0.32,
"power_watt":
2.38,
"robot_status":
"ONLINE",
"location":
"LAB",
"timestamp":
"2026-07-18T07:30:00"
}
13. AI Agent Logic
The AI Agent receives the emotion information and produces an intelligent response.
AI Agent Input
Emotion: SAD Confidence: 89% Battery: 7.42 V Power: 2.38 W
AI Agent Prompt
You are an intelligent emotional interaction robot.
Analyze the following emotion:
Emotion: {{$json.emotion}}
Confidence: {{$json.confidence}}
Generate:
1. A short supportive response.
2. A robot action.
3. An alert level.
4. A suggested voice tone.
Return JSON only.
AI Agent Output
{
"message":
"You seem a little sad. I am here to talk with you.",
"robot_action":
"COMFORT_MODE",
"alert_level":
"MEDIUM",
"voice_tone":
"CALM"
}
14. n8n Workflow Architecture
+------------------+
| ESP32 Webhook |
+--------+---------+
|
v
+------------------+
| JSON Parser |
+--------+---------+
|
v
+------------------+
| AI Agent |
| Emotion Analysis |
+--------+---------+
|
v
+------------------+
| IF Node |
| Emotion Type |
+----+--------+----+
| |
v v
Happy Sad
| |
+--------+---------+
|
v
+----------------+
| Telegram Alert |
+----------------+
|
v
+----------------+
| Voice Message |
+----------------+
|
v
+----------------+
| Google Sheets |
+----------------+
|
v
+----------------+
| ThingSpeak |
+----------------+
|
v
+----------------+
| Web Dashboard |
+----------------+
15. n8n Workflow JSON
{
"name":
"AI Emotion Recognition Robot",
"nodes": [
{
"parameters": {
"httpMethod":
"POST",
"path":
"emotion-robot",
"responseMode":
"onReceived"
},
"name":
"ESP32 Webhook",
"type":
"n8n-nodes-base.webhook"
},
{
"parameters": {
"jsCode":
"const data = $json.body || $json;
return [{
json: {
device_id:
data.device_id,
emotion:
data.emotion,
confidence:
data.confidence,
voltage:
data.voltage,
current:
data.current,
power:
data.power,
robot_status:
data.robot_status,
timestamp:
new Date().toISOString()
}
}];"
},
"name":
"Process Emotion Data",
"type":
"n8n-nodes-base.code"
},
{
"parameters": {
"method":
"POST",
"url":
"YOUR_AI_AGENT_ENDPOINT",
"sendBody":
true,
"specifyBody":
"json",
"jsonBody":
"={
\"emotion\":
\"{{$json.emotion}}\",
\"confidence\":
\"{{$json.confidence}}\",
\"power\":
\"{{$json.power}}\"
}"
},
"name":
"AI Emotion Agent",
"type":
"n8n-nodes-base.httpRequest"
},
{
"parameters": {
"operation":
"append",
"documentId":
"YOUR_GOOGLE_SHEET_ID",
"sheetName":
"EmotionLogs"
},
"name":
"Google Sheets Log",
"type":
"n8n-nodes-base.googleSheets"
},
{
"parameters": {
"chatId":
"YOUR_TELEGRAM_CHAT_ID",
"text":
"🤖 Emotion Robot Alert
Emotion:
{{$json.emotion}}
Confidence:
{{$json.confidence}}
Power:
{{$json.power}} W"
},
"name":
"Telegram Alert",
"type":
"n8n-nodes-base.telegram"
}
]
}
16. Telegram Bot Setup
Open Telegram and search for BotFather.
/newbot
Create a bot name such as:
Emotion Robot AI Bot
Telegram generates a bot token.
BOT_TOKEN
The token must be kept private.
The Telegram bot is used for:
- Emotion alerts
- Robot status alerts
- Battery alerts
- Power consumption alerts
- AI voice notifications
17. Telegram Voice Notification Automation
Emotion Detected
|
v
AI Generates Message
|
v
Text-to-Speech
|
v
Audio File
|
v
Telegram Voice Message
Example:
Emotion: SAD AI Message: "You seem a little sad. I am here to support you."
18. Google Sheets Integration
Create a Google Sheet named:
EmotionLogs
| Timestamp | Device ID | Emotion | Confidence | Temperature | Battery | Current | Power | AI Response |
|---|---|---|---|---|---|---|---|---|
| 2026-07-18 | EMOTION_ROBOT_001 | Happy | 94% | 29.5 °C | 7.42 V | 0.32 A | 2.38 W | You look happy today! |
19. ThingSpeak Cloud Dashboard
Recommended ThingSpeak fields:
Field 1: Emotion Code Field 2: Emotion Confidence Field 3: Temperature Field 4: Humidity Field 5: Battery Voltage Field 6: Current Field 7: Power Consumption Field 8: Robot Status
Emotion code example:
1 = Happy 2 = Sad 3 = Angry 4 = Neutral 5 = Fear 6 = Surprise
20. PHP IoT Web Dashboard
<?php
$emotion = "HAPPY";
$confidence = 94;
$temperature = 29.5;
$humidity = 58;
$battery = 7.42;
$power = 2.38;
$status = "ONLINE";
?>
<!DOCTYPE html>
<html>
<head>
<title>
AI Emotion Recognition Robot
</title>
</head>
<body>
<h1>
🤖 AI Emotion Recognition Robot
</h1>
<p>
Current Emotion:
<?php echo $emotion; ?>
</p>
<p>
Confidence:
<?php echo $confidence; ?>%
</p>
<p>
Temperature:
<?php echo $temperature; ?> °C
</p>
<p>
Humidity:
<?php echo $humidity; ?>%
</p>
<p>
Battery:
<?php echo $battery; ?> V
</p>
<p>
Power:
<?php echo $power; ?> W
</p>
<p>
Robot Status:
<?php echo $status; ?>
</p>
</body>
</html>
21. Power Consumption Monitoring
Battery | v INA219 Sensor | v ESP32 | v Power Calculation
Power is calculated using voltage and current measurement.
Power = Voltage x Current
Example:
Voltage = 7.42 V Current = 0.32 A Power = 7.42 x 0.32 Power = 2.37 W
22. AI Power Consumption Prediction
Historical power data is stored in Google Sheets or a cloud database. The AI system analyzes the historical data to predict future power consumption.
Time Power 08:00 2.1 W 09:00 2.4 W 10:00 2.8 W 11:00 3.1 W
The AI Agent can identify increasing power consumption and generate an alert.
Power usage is increasing. Expected power consumption in the next hour = 3.4 W
23. Power Prediction Flowchart
+----------------+
| Power Sensor |
+-------+--------+
|
v
+----------------+
| Collect Data |
+-------+--------+
|
v
+----------------+
| Store History |
+-------+--------+
|
v
+----------------+
| AI Analysis |
+-------+--------+
|
v
+----------------+
| Predict Power |
+-------+--------+
|
v
+----------------+
| Alert if High |
+----------------+
24. Complete Data Flow
HUMAN
|
v
+-------------+
| Camera |
| Microphone |
+------+------+
|
v
+-------------+
| AI Emotion |
| Recognition |
+------+------+
|
v
+-------------+
| ESP32 |
| Controller |
+------+------+
|
v
+-------------+
| Wi-Fi |
+------+------+
|
v
+-------------+
| n8n |
| Automation |
+------+------+
|
+----+----+---------+---------+
| | | |
v v v v
AI Telegram Google ThingSpeak
Agent Voice Sheets Cloud
Alert
|
v
+---------------+
| Robot Response |
+---------------+
25. Real-Time Example
Scenario: User Shows a Sad Expression
Camera: Emotion = SAD Confidence = 89%
ESP32 sends:
{
"emotion": "sad",
"confidence": 0.89
}
AI Agent generates:
"You seem a little sad. I am here to talk with you."
Robot action:
Movement = Slow Head Movement Voice = Calm LED = Sad Indication
Telegram alert:
🤖 Emotion Alert Emotion: SAD Confidence: 89% AI Action: COMFORT_MODE
26. Advantages
- Human-friendly robotic interaction
- Real-time emotion recognition
- AI-powered decision making
- IoT cloud connectivity
- Automated Telegram notifications
- Voice alerts
- Google Sheets data storage
- ThingSpeak visualization
- Power consumption analysis
- Expandable architecture
- Remote monitoring
- Suitable for academic research
27. Applications
- Healthcare assistance
- Elderly care
- Education
- Smart homes
- Social robotics
- Customer service
- Human-robot interaction research
- Emotional wellness support
This system should not be considered a medical diagnostic device.
28. Future Enhancements
- Multi-language voice interaction
- Advanced facial emotion recognition
- Voice emotion recognition
- Human identity recognition
- Personalized AI behavior
- Long-term emotion history analysis
- Reinforcement learning
- Edge AI processing
- Autonomous navigation
- Object recognition
- Human activity recognition
- Mobile application
- Mobile push notifications
- Cloud-based AI model training
- Battery health prediction
- Solar charging
- Multi-robot communication
- Offline AI operation
- Advanced speech recognition
- Personalized emotional memory
29. Deployment Guide
Hardware Deployment
- Assemble ESP32 circuit.
- Connect camera module.
- Connect sensors.
- Connect motor driver.
- Connect audio module.
- Connect battery.
- Test each component individually.
Software Deployment
- Install Arduino IDE.
- Install ESP32 Board Package.
- Install required libraries.
- Upload ESP32 firmware.
- Configure Wi-Fi.
- Create Telegram Bot.
- Configure n8n.
- Create Google Sheet.
- Create ThingSpeak channel.
- Deploy PHP dashboard.
- Test complete system.
30. Testing Procedure
ESP32 Testing
✓ ESP32 boots ✓ Wi-Fi connects ✓ IP address received
Sensor Testing
✓ Camera works ✓ Temperature sensor works ✓ Power sensor works ✓ Ultrasonic sensor works
Emotion Testing
Happy Sad Angry Neutral Surprise
n8n Testing
✓ Webhook receives data ✓ AI Agent responds ✓ Telegram notification works ✓ Google Sheets receives data ✓ ThingSpeak receives data
31. Final Project Summary
The AI-Based Emotion Recognition Robot for Human Interaction is a complete AI-powered Agentic IoT project that combines embedded systems, artificial intelligence, robotics, automation and cloud services.
AI Emotion Recognition
+
ESP32 IoT Controller
+
AI Agent
+
n8n Automation
+
Telegram Voice Alerts
+
Google Sheets
+
ThingSpeak Cloud
+
PHP IoT Dashboard
+
Power Prediction
The complete system follows the intelligent process:
SENSE → UNDERSTAND → DECIDE → RESPOND → NOTIFY → RECORD → PREDICT
No comments:
Post a Comment