AI Eye-Blink Controlled Virtual Keyboard + Agentic ESP32 IoT + n8n + Telegram Voice Alerts + Google Sheets + ThingSpeak + Web Dashboard
The project can be presented as an AI-assisted accessibility and IoT monitoring system.
The Raspberry Pi performs the computer-vision/eye-blink interaction, while the ESP32 performs IoT sensing and device control. n8n acts as the automation/orchestration layer, an AI Agent interprets events and decides what action should happen, and Telegram/Google Sheets/ThingSpeak/Web Dashboard provide notification, logging and visualization.
1. Project Title
AI-Powered Eye Blink Controlled Virtual Keyboard and Agentic IoT Monitoring System Using Raspberry Pi, ESP32, Deep Learning, n8n, Telegram, Google Sheets and ThingSpeak
Short title
AI Eye-Blink Keyboard + Agentic ESP32 IoT
2. Abstract
This project proposes an assistive human-computer interaction and IoT automation system that allows a user to operate a virtual keyboard using eye-blink gestures instead of a physical keyboard or mouse.
A camera connected to a Raspberry Pi captures the user's face. A deep-learning-based facial landmark model identifies the eye region. Eye openness is continuously analyzed to detect intentional blinks while rejecting normal eye movements as much as possible.
The detected blink becomes an input command for a virtual keyboard. A scanning mechanism moves through keyboard rows and columns, allowing the user to select characters using intentional blinks.
The Raspberry Pi is additionally connected to an ESP32-based IoT subsystem. The ESP32 collects sensor information such as temperature, humidity, gas/smoke level, motion or other selected parameters and sends the data through Wi-Fi.
An n8n automation server receives the IoT data through a webhook. It stores measurements in Google Sheets, updates ThingSpeak, sends notifications through Telegram and passes important events to an AI Agent. The AI Agent analyzes the event and determines whether a normal notification, warning or emergency alert should be generated.
For critical conditions, the automation workflow can generate a Telegram voice notification, allowing a remote user/caregiver to receive an audible alert.
The system therefore combines:
- Computer vision
- Deep learning
- Human-computer interaction
- Assistive technology
- Raspberry Pi
- ESP32
- IoT
- AI agents
- n8n workflow automation
- Telegram
- Google Sheets
- ThingSpeak
- Web dashboard
3. Main Objective
The main objective is to build a system in which:
Eye blink → virtual keyboard command → Raspberry Pi → IoT/AI event → n8n → AI Agent → cloud logging → dashboard → Telegram notification/voice alert
The system can be adapted for:
- Assistive communication
- Elderly monitoring
- Patient-room monitoring
- Smart-home automation
- Industrial safety monitoring
- Remote environmental monitoring
- Emergency alert systems
4. Overall System Architecture
┌─────────────────────┐
│ USER │
│ │
│ Eye Blink Gesture │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ USB/CSI CAMERA │
└──────────┬──────────┘
│ Video
▼
┌─────────────────────────────┐
│ RASPBERRY PI │
│ │
│ Face Detection │
│ ↓ │
│ Facial Landmarks │
│ ↓ │
│ Eye/Blink Detection │
│ ↓ │
│ Blink Gesture Classifier │
│ ↓ │
│ Virtual Keyboard │
└─────────────┬───────────────┘
│
HTTP/MQTT/Webhook
│
▼
┌────────────────────────┐
│ n8n SERVER │
│ │
│ Webhook │
│ ↓ │
│ Data Validation │
│ ↓ │
│ AI Agent │
│ ↓ │
│ Decision/Automation │
└──────┬───────┬─────────┘
│ │
┌────────────┘ └─────────────┐
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Google Sheets │ │ ThingSpeak │
│ │ │ │
│ Historical Data │ │ IoT Graphs │
└─────────────────┘ └─────────────────┘
│
▼
┌────────────────┐
│ Telegram │
│ │
│ Text Alert │
│ Voice Alert │
└────────────────┘
┌──────────────────────────────────────┐
│ ESP32 │
│ │
│ Temperature ─┐ │
│ Humidity ─────┤ │
│ Gas ──────────┤ │
│ Motion ───────┤ → Wi-Fi → n8n │
│ Emergency ────┘ │
└──────────────────────────────────────┘
The ESP32 can operate independently from the Raspberry Pi, while n8n provides the common automation layer.
5. Recommended Hardware
Raspberry Pi section
| Component | Quantity |
|---|---|
| Raspberry Pi 4/5 | 1 |
| Raspberry Pi Camera Module or USB webcam | 1 |
| MicroSD card | 1 |
| 5 V Raspberry Pi power supply | 1 |
| Monitor | 1 |
| Keyboard for initial setup | 1 |
| Optional speaker | 1 |
Raspberry Pi's current documentation supports Python GPIO control through GPIO Zero, and Raspberry Pi's newer AI hardware can also accelerate neural-network workloads.
For a student prototype, Raspberry Pi 5 + Camera Module 3/USB camera is a good starting point. An AI accelerator is optional rather than mandatory for the basic blink system.
6. ESP32 Hardware
| Component | Quantity |
|---|---|
| ESP32 DevKit | 1 |
| DHT22 temperature/humidity sensor | 1 |
| MQ-2 gas/smoke sensor | 1 |
| PIR motion sensor | 1 |
| Buzzer | 1 |
| LED | 1 |
| Push button | 1 |
| 220 Ω resistor | 1 |
| Breadboard | 1 |
| Jumper wires | Several |
You can replace the sensors with:
- BMP280
- BME280
- LDR
- Soil-moisture sensor
- Flame sensor
- Current sensor
- Pulse/heart sensor
- Door sensor
- Vibration sensor
ESP32 supports Wi-Fi station mode for connecting to an access point and Internet-based IoT applications.
7. Why Use Raspberry Pi + ESP32?
They perform different jobs.
Raspberry Pi
Best for:
- Camera
- Computer vision
- Deep learning
- Python
- Virtual keyboard
- Local web application
- Higher-level processing
ESP32
Best for:
- Sensors
- Wi-Fi
- GPIO
- Low-cost IoT
- Real-time device control
- Remote monitoring
n8n
Best for:
- Workflow automation
- API integration
- AI Agent orchestration
- Telegram
- Google Sheets
- Webhooks
- Cloud services
Therefore:
Raspberry Pi = AI/Computer Vision
ESP32 = IoT/Hardware
n8n = Automation/Agent
Cloud = Storage/Dashboard
Telegram = Human Notification
8. Raspberry Pi Eye-Blink System
8.1 Camera
The camera continuously captures frames.
Camera
│
▼
Frame
│
▼
Face Detection
│
▼
Facial Landmark Detection
│
▼
Eye Landmark Extraction
│
▼
Eye Aspect Ratio
│
▼
Blink Detection
Raspberry Pi cameras use the CSI camera interface on supported models, while USB cameras can also be used.
9. Deep Learning Component
A good architecture is:
Camera
↓
Deep-learning facial landmark model
↓
Face landmarks
↓
Eye landmarks
↓
Geometric eye features
↓
Temporal blink classifier
↓
Keyboard command
The deep-learning portion performs the difficult visual task of locating facial/eye landmarks.
The actual blink decision can then use a lightweight temporal algorithm.
This is often preferable to putting a large neural network directly in the keyboard-control loop.
10. Eye Aspect Ratio
One common measurement is the Eye Aspect Ratio (EAR).
For six eye landmarks:
P2 -------- P3
/ \
P1 --- --- P4
\ /
P6 -------- P5
EAR can be represented as:
When the eye is open:
EAR ≈ high
When the eye closes:
EAR ↓
A blink can therefore be detected as:
OPEN → CLOSED → OPEN
rather than simply checking whether the eye is currently closed.
This distinction is important because the user may intentionally hold their eyes closed.
11. Blink State Machine
Use a state machine:
EAR < threshold
OPEN ─────────────────────────► CLOSING
│ │
│ │
│ ▼
│ CLOSED
│ │
│ │
└──────── EAR > threshold ◄──────┘
│
▼
BLINK
The software should additionally require:
- Minimum closed duration
- Maximum closed duration
- Cooldown period
- Confidence threshold
This reduces accidental selections.
12. Virtual Keyboard
A normal keyboard would be difficult to operate with only blinking.
Therefore, use automatic scanning.
Example:
+--------------------------------+
| A | B | C | D | E | F | G | H |
+--------------------------------+
| I | J | K | L | M | N | O | P |
+--------------------------------+
| Q | R | S | T | U | V | W | X |
+--------------------------------+
| Y | Z | 0 | 1 | 2 | 3 | 4 | 5 |
+--------------------------------+
| SPACE | BACKSPACE | ENTER | SOS|
+--------------------------------+
The highlighted selection moves automatically.
13. Two-Level Blink Selection
A very practical control scheme is:
Step 1 — Row selection
The system scans:
ROW 1
ROW 2
ROW 3
ROW 4
ROW 5
User blinks when the desired row is highlighted.
Step 2 — Column selection
The system then scans the keys in that row.
User blinks again.
Example
User wants:
H
System:
Row 1 highlighted
↓
User blink
↓
Row 1 selected
↓
A → B → C → D → E → F → G → H
↑
User blink
↓
"H"
This makes the keyboard usable without gaze tracking hardware.
14. Special Commands
Reserve several keys:
SPACE
BACKSPACE
ENTER
CLEAR
SOS
MODE
For example:
Double blink
Double blink → ENTER
Long blink
Long blink → SOS
However, these should be configurable because different users have different blink capabilities.
15. Example User Interaction
Screen:
┌────────────────────────────────────────┐
│ Message: HELLO │
│ │
│ [A] B C D E F G H │
│ I J K L M N O P │
│ Q R S T U V W X │
│ Y Z 0 1 2 3 4 5 │
│ │
│ STATUS: ROW SCANNING │
└────────────────────────────────────────┘
Blink:
A
Then:
H
Then:
E
etc.
The message is assembled automatically.
16. Raspberry Pi Software Architecture
Python Application
│
├── Camera
│
├── Face Landmark Model
│
├── Blink Detector
│
├── Keyboard Scanner
│
├── Message Buffer
│
├── Event Manager
│
└── HTTP Client
│
▼
n8n
17. Raspberry Pi Software Installation
Install Raspberry Pi OS and update it.
sudo apt update
sudo apt full-upgrade -y
Install Python tools:
sudo apt install -y python3-pip python3-venv python3-opencv
Create project:
mkdir ~/ai_blink_iot
cd ~/ai_blink_iot
python3 -m venv venv
source venv/bin/activate
Install Python packages:
pip install numpy
pip install opencv-python
pip install requests
For the facial-landmark model, install the appropriate current MediaPipe package/model for your Raspberry Pi OS/Python version.
18. Project Directory
Use this structure:
ai_blink_iot/
│
├── main.py
├── blink_detector.py
├── keyboard.py
├── camera.py
├── config.py
├── n8n_client.py
├── utils.py
│
├── models/
│ └── face_landmarker.task
│
├── web/
│ ├── index.html
│ ├── style.css
│ └── app.js
│
├── data/
│ └── messages.json
│
└── logs/
19. Blink Detection Code
Below is a simplified implementation of the EAR logic.
import time
import numpy as np
class BlinkDetector:
def __init__(
self,
threshold=0.21,
min_closed=0.08,
max_closed=0.8,
cooldown=0.6
):
self.threshold = threshold
self.min_closed = min_closed
self.max_closed = max_closed
self.cooldown = cooldown
self.closed_start = None
self.last_blink = 0
def distance(self, a, b):
return np.linalg.norm(np.array(a) - np.array(b))
def ear(self, points):
p1, p2, p3, p4, p5, p6 = points
vertical_1 = self.distance(p2, p6)
vertical_2 = self.distance(p3, p5)
horizontal = self.distance(p1, p4)
if horizontal == 0:
return 0
return (vertical_1 + vertical_2) / (2.0 * horizontal)
def update(self, eye_points):
value = self.ear(eye_points)
now = time.monotonic()
blink = False
if value < self.threshold:
if self.closed_start is None:
self.closed_start = now
else:
if self.closed_start is not None:
duration = now - self.closed_start
if (
self.min_closed <= duration <= self.max_closed
and now - self.last_blink > self.cooldown
):
blink = True
self.last_blink = now
self.closed_start = None
return blink, value
20. Why This Code Is Useful
The detector doesn't simply say:
eye closed = key press
Instead:
eye closes
↓
start timer
↓
eye opens
↓
calculate duration
↓
if duration valid
↓
BLINK EVENT
This prevents a continuous eye closure from generating hundreds of key presses.
21. Keyboard Scanner Code
import time
class VirtualKeyboard:
def __init__(self):
self.rows = [
list("ABCDEFGH"),
list("IJKLMNOP"),
list("QRSTUVWX"),
list("YZ012345"),
["SPACE", "BACKSPACE", "ENTER", "SOS"]
]
self.row_index = 0
self.key_index = 0
self.mode = "ROW"
self.message = ""
def current_item(self):
if self.mode == "ROW":
return self.row_index
return self.rows[self.row_index][self.key_index]
def blink(self):
if self.mode == "ROW":
# Select current row
self.mode = "KEY"
self.key_index = 0
else:
key = self.rows[self.row_index][self.key_index]
self.execute(key)
self.mode = "ROW"
def scan(self):
if self.mode == "ROW":
self.row_index += 1
if self.row_index >= len(self.rows):
self.row_index = 0
else:
self.key_index += 1
if self.key_index >= len(self.rows[self.row_index]):
self.key_index = 0
def execute(self, key):
if key == "SPACE":
self.message += " "
elif key == "BACKSPACE":
self.message = self.message[:-1]
elif key == "ENTER":
print("MESSAGE:", self.message)
elif key == "SOS":
print("SOS EVENT")
else:
self.message += key
22. Main Raspberry Pi Program
Conceptually:
while True:
frame = camera.read()
landmarks = detect_face_landmarks(frame)
if landmarks:
eye_points = extract_eye_points(landmarks)
blink, ear = blink_detector.update(eye_points)
if blink:
keyboard.blink()
event = {
"type": "blink",
"ear": ear,
"message": keyboard.message
}
send_to_n8n(event)
keyboard.scan()
display_keyboard(frame)
The exact camera/model integration depends on whether you use a USB webcam, Camera Module, MediaPipe Tasks, or another landmark model.
23. Raspberry Pi → n8n Communication
The Raspberry Pi can send:
{
"device": "raspberry_pi",
"event": "blink_keyboard",
"message": "HELLO",
"command": "ENTER",
"timestamp": "2026-08-18T20:30:00"
}
using HTTP POST.
Example Python:
import requests
N8N_URL = "https://YOUR-N8N-DOMAIN/webhook/ai-keyboard"
def send_to_n8n(data):
try:
response = requests.post(
N8N_URL,
json=data,
timeout=10
)
return response.status_code == 200
except requests.RequestException as e:
print("n8n error:", e)
return False
24. ESP32 IoT Architecture
The ESP32 reads sensors.
┌─────────────┐
│ DHT22 │
└──────┬──────┘
│
┌──────▼──────┐
│ ESP32 │
│ │
MQ-2 ───────►│ ADC │
PIR ────────►│ GPIO │
Button ─────►│ GPIO │
│ │
│ Wi-Fi │
└──────┬──────┘
│
HTTP
│
▼
n8n
25. ESP32 Wiring
A possible configuration:
| Device | ESP32 GPIO |
|---|---|
| DHT22 DATA | GPIO 4 |
| MQ-2 analog | GPIO 34 |
| PIR OUT | GPIO 27 |
| Buzzer | GPIO 25 |
| LED | GPIO 2 |
| SOS Button | GPIO 26 |
| GND | GND |
| VCC | 3.3/5 V according to sensor |
Important: ESP32 GPIOs are not 5-V tolerant. Ensure that sensor outputs connected to ESP32 pins remain within the permitted voltage range. For modules with 5-V analog outputs, use appropriate level shifting/division.
No comments:
Post a Comment