Initial commit: LogMaster with Viewer Collector Monitor modes
This commit is contained in:
377
docs/MONITOR.md
Normal file
377
docs/MONITOR.md
Normal file
@@ -0,0 +1,377 @@
|
||||
# 🔍 Log Monitor — User Guide
|
||||
|
||||
## Overview
|
||||
|
||||
The Log Monitor scans log files on a schedule, matches regex patterns, and sends alerts via Telegram, Webhooks, Microsoft Teams, or Email. It supports time-based filtering, scheduled scans (daily, hourly, continuous), and automatic attachment of full results when matches exceed a threshold.
|
||||
|
||||
---
|
||||
|
||||
## Starting the Monitor
|
||||
|
||||
### Option 1: Environment File
|
||||
```bash
|
||||
cp .env.monitor .env
|
||||
node dist-server/server.js
|
||||
```
|
||||
|
||||
### Option 2: Inline
|
||||
```bash
|
||||
LOG_MODE=monitor PORT=8082 MONITOR_SCAN_INTERVAL=30000 node dist-server/server.js
|
||||
```
|
||||
|
||||
### Option 3: Docker
|
||||
```bash
|
||||
docker run -d -p 8082:8080 \
|
||||
-e LOG_MODE=monitor \
|
||||
-e MONITOR_SCAN_INTERVAL=30000 \
|
||||
-v /var/log:/logs:ro \
|
||||
logviewer
|
||||
```
|
||||
|
||||
### Option 4: Config File
|
||||
```json
|
||||
{
|
||||
"mode": "monitor",
|
||||
"port": 8082,
|
||||
"monitor": {
|
||||
"enabled": true,
|
||||
"scanIntervalMs": 30000,
|
||||
"rules": []
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
|
||||
│ Log Files │────▶│ Scan Engine │────▶│ Pattern Match │
|
||||
│ (sources) │ │ (interval) │ │ (regex) │
|
||||
└─────────────┘ └──────────────┘ └────────┬────────┘
|
||||
│
|
||||
matches found?
|
||||
│
|
||||
┌──────────────┴──────────────┐
|
||||
│ │
|
||||
≤ maxLines > maxLines
|
||||
│ │
|
||||
┌─────▼─────┐ ┌───────▼───────┐
|
||||
│ Inline │ │ Inline (N) │
|
||||
│ in alert │ │ + .gz attach │
|
||||
└─────┬─────┘ └───────┬───────┘
|
||||
│ │
|
||||
└──────────────┬──────────────┘
|
||||
│
|
||||
┌──────────────▼──────────────┐
|
||||
│ Send Notifications │
|
||||
│ Telegram │ Webhook │ Teams │
|
||||
└─────────────────────────────┘
|
||||
```
|
||||
|
||||
1. **Scan** — Every N seconds (configurable), the monitor reads all source files
|
||||
2. **Filter** — Only files matching the time range are scanned (today, this week, etc.)
|
||||
3. **Match** — Each line is tested against the configured regex patterns
|
||||
4. **Alert** — If matches are found AND cooldown has expired, notifications are sent
|
||||
5. **Attach** — If matches exceed `maxLinesInMessage`, full results are gzipped and attached
|
||||
|
||||
---
|
||||
|
||||
## GUI Guide
|
||||
|
||||
### Layout
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ 🔍 Log Monitor 2 active rules · 3 alerts│
|
||||
├──────────────┬──────────────────────────────────────────────┤
|
||||
│ │ │
|
||||
│ 📋 Rules │ Rule Editor / Alert Detail │
|
||||
│ ┌──────────┐│ │
|
||||
│ │●Root DB ││ [General] │
|
||||
│ │ 2 src ││ Name: Root DB Access Detection │
|
||||
│ │ 6 match ││ Enabled: Yes │
|
||||
│ ├──────────┤│ │
|
||||
│ │●Failed ││ [Sources] │
|
||||
│ │ 1 src ││ /var/log/db-audit.log │
|
||||
│ │ 4 match ││ /var/log/db-audit-*.log.gz │
|
||||
│ └──────────┘│ │
|
||||
│ │ [Patterns] │
|
||||
│ 📊 Alerts │ root.*login │
|
||||
│ ┌──────────┐│ GRANT.*PRIVILEGES │
|
||||
│ │🚨 Root DB││ DROP TABLE │
|
||||
│ │ 6 matches││ │
|
||||
│ │ 10:30:01 ││ [Schedule] │
|
||||
│ ├──────────┤│ Mode: Daily at 07:00 │
|
||||
│ │🚨 Failed ││ │
|
||||
│ │ 4 matches││ [Notifications] │
|
||||
│ │ 10:30:01 ││ [TELEGRAM] Bot: 811... Chat: 859... │
|
||||
│ └──────────┘│ [WEBHOOK] https://hooks.slack.com/... │
|
||||
│ │ │
|
||||
│ │ [💾 Save] [🧪 Test Now] [🗑 Delete] │
|
||||
└──────────────┴──────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Creating a Rule
|
||||
|
||||
1. Click **"+ New"** in the sidebar
|
||||
2. Fill in the fields:
|
||||
- **Name** — descriptive name for the rule
|
||||
- **Sources** — one file path per line (supports globs like `/var/log/*.log`)
|
||||
- **Time Range** — which files to scan (today, week, month, last N hours, all)
|
||||
- **Patterns** — one regex per line
|
||||
- **Match Logic** — "any" (OR) or "all" (AND)
|
||||
- **Schedule** — continuous, daily at time, or every N hours
|
||||
- **Max Lines** — how many lines to include inline in notifications
|
||||
- **Cooldown** — minimum time between alerts (prevents spam)
|
||||
3. Add notification channels
|
||||
4. Click **"💾 Save Rule"**
|
||||
|
||||
### Testing a Rule
|
||||
|
||||
1. Select a rule from the sidebar
|
||||
2. Click **"🧪 Test Now"**
|
||||
3. The rule runs immediately and shows results below the form
|
||||
4. This does NOT respect cooldown — always runs
|
||||
|
||||
### Testing Notifications
|
||||
|
||||
1. In the Notifications section, configure a channel
|
||||
2. Click **"🧪 Test Notification"**
|
||||
3. A test message is sent to verify the channel works
|
||||
|
||||
### Viewing Alerts
|
||||
|
||||
- Alerts appear in the **"📊 Recent Alerts"** section of the sidebar
|
||||
- Click an alert to see the full detail view with all matched lines
|
||||
- Lines are color-coded: red for ERROR/FATAL, orange for WARN
|
||||
|
||||
---
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `LOG_MODE` | `both` | Set to `monitor` |
|
||||
| `PORT` | `8080` | HTTP port |
|
||||
| `MONITOR_ENABLED` | `true` | Enable/disable scanning |
|
||||
| `MONITOR_SCAN_INTERVAL` | `60000` | Scan interval in ms |
|
||||
| `MONITOR_STORAGE_DIR` | `./monitor-data` | Where to store state |
|
||||
| `MONITOR_RULES` | `[]` | JSON array of rules (see below) |
|
||||
|
||||
### Rule Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "unique-id",
|
||||
"name": "Human-readable name",
|
||||
"enabled": true,
|
||||
"sources": ["/path/to/logs/*.log"],
|
||||
"patterns": ["regex1", "regex2"],
|
||||
"patternLogic": "any",
|
||||
"timeRange": { "type": "week", "hours": 24 },
|
||||
"schedule": { "mode": "daily", "value": "07:00", "timezone": "Europe/Berlin" },
|
||||
"notifications": [
|
||||
{ "type": "telegram", "botToken": "123:ABC", "chatId": "-100123" }
|
||||
],
|
||||
"maxLinesInMessage": 20,
|
||||
"cooldownMs": 300000,
|
||||
"matchCount": 0,
|
||||
"lastTriggered": null
|
||||
}
|
||||
```
|
||||
|
||||
### Time Range Options
|
||||
|
||||
| Type | Description | Use Case |
|
||||
|------|-------------|----------|
|
||||
| `today` | Only files modified today | Daily reports |
|
||||
| `week` | Files from last 7 days | Weekly audits |
|
||||
| `month` | Files from last 30 days | Monthly reviews |
|
||||
| `hours` | Files from last N hours | Recent activity |
|
||||
| `all` | All files regardless of age | Full history scan |
|
||||
|
||||
### Schedule Options
|
||||
|
||||
| Mode | Value | Description |
|
||||
|------|-------|-------------|
|
||||
| `interval` | — | Runs every scan interval (continuous monitoring) |
|
||||
| `daily` | `07:00` | Runs once per day at the specified time |
|
||||
| `hourly` | `2` | Runs every N hours |
|
||||
|
||||
### Notification Channels
|
||||
|
||||
**Telegram:**
|
||||
```json
|
||||
{ "type": "telegram", "botToken": "123456:ABC-DEF...", "chatId": "8599028500" }
|
||||
```
|
||||
|
||||
**Webhook (Slack, Discord, custom):**
|
||||
```json
|
||||
{ "type": "webhook", "url": "https://hooks.slack.com/services/T.../B.../..." }
|
||||
```
|
||||
|
||||
**Microsoft Teams:**
|
||||
```json
|
||||
{ "type": "teams", "url": "https://outlook.office.com/webhook/..." }
|
||||
```
|
||||
|
||||
**Email (SMTP):**
|
||||
```json
|
||||
{ "type": "email", "to": "admin@company.com", "smtpHost": "smtp.gmail.com", "smtpPort": 587, "smtpUser": "alerts@company.com", "smtpPass": "app-password" }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example Configurations
|
||||
|
||||
### 1. Real-time Root Login Detection
|
||||
|
||||
Scans every 10 seconds, alerts immediately via Telegram:
|
||||
```env
|
||||
LOG_MODE=monitor
|
||||
PORT=8082
|
||||
MONITOR_SCAN_INTERVAL=10000
|
||||
MONITOR_RULES=[{"id":"root-login","name":"Root Login Alert","enabled":true,"sources":["/var/log/auth.log"],"patterns":["session opened for user root","su.*root","sudo.*root"],"patternLogic":"any","timeRange":{"type":"today"},"schedule":{"mode":"interval"},"notifications":[{"type":"telegram","botToken":"YOUR_BOT_TOKEN","chatId":"YOUR_CHAT_ID"}],"maxLinesInMessage":10,"cooldownMs":60000,"matchCount":0,"lastTriggered":null}]
|
||||
```
|
||||
|
||||
### 2. Daily DB Audit Report at 7:00 AM
|
||||
|
||||
Scans once per day, sends full report:
|
||||
```env
|
||||
LOG_MODE=monitor
|
||||
PORT=8082
|
||||
MONITOR_SCAN_INTERVAL=60000
|
||||
MONITOR_RULES=[{"id":"db-audit","name":"Daily DB Audit","enabled":true,"sources":["/var/log/postgresql/audit.log","/var/log/mysql/audit*.log"],"patterns":["admin.*LOGIN","GRANT","DROP","ALTER.*TABLE","CREATE USER"],"patternLogic":"any","timeRange":{"type":"today"},"schedule":{"mode":"daily","value":"07:00","timezone":"Europe/Berlin"},"notifications":[{"type":"telegram","botToken":"TOKEN","chatId":"CHAT_ID"},{"type":"webhook","url":"https://hooks.slack.com/services/..."}],"maxLinesInMessage":20,"cooldownMs":86400000,"matchCount":0,"lastTriggered":null}]
|
||||
```
|
||||
|
||||
### 3. Error Spike Detection (every 5 minutes)
|
||||
|
||||
```env
|
||||
LOG_MODE=monitor
|
||||
PORT=8082
|
||||
MONITOR_SCAN_INTERVAL=30000
|
||||
MONITOR_RULES=[{"id":"error-spike","name":"Error Spike","enabled":true,"sources":["/app/logs/production.log"],"patterns":["\\bERROR\\b","\\bFATAL\\b","Exception","panic:"],"patternLogic":"any","timeRange":{"type":"hours","hours":1},"schedule":{"mode":"hourly","value":"0.083"},"notifications":[{"type":"teams","url":"https://outlook.office.com/webhook/..."}],"maxLinesInMessage":30,"cooldownMs":300000,"matchCount":0,"lastTriggered":null}]
|
||||
```
|
||||
|
||||
### 4. Security: Unauthorized Access Attempts
|
||||
|
||||
```env
|
||||
MONITOR_RULES=[{"id":"unauth","name":"Unauthorized Access","enabled":true,"sources":["/var/log/nginx/access.log","/var/log/nginx/error.log"],"patterns":["403 Forbidden","401 Unauthorized","\\.\\./\\.\\./","etc/passwd","<script>"],"patternLogic":"any","timeRange":{"type":"today"},"schedule":{"mode":"interval"},"notifications":[{"type":"telegram","botToken":"TOKEN","chatId":"CHAT"},{"type":"webhook","url":"https://siem.internal/alerts"}],"maxLinesInMessage":15,"cooldownMs":120000,"matchCount":0,"lastTriggered":null}]
|
||||
```
|
||||
|
||||
### 5. Multiple Rules via Config File
|
||||
|
||||
```json
|
||||
{
|
||||
"mode": "monitor",
|
||||
"port": 8082,
|
||||
"monitor": {
|
||||
"enabled": true,
|
||||
"scanIntervalMs": 30000,
|
||||
"rules": [
|
||||
{
|
||||
"id": "root-access",
|
||||
"name": "Root DB Access",
|
||||
"enabled": true,
|
||||
"sources": ["/var/log/db-audit.log"],
|
||||
"patterns": ["root.*login", "GRANT", "DROP TABLE"],
|
||||
"patternLogic": "any",
|
||||
"timeRange": { "type": "today" },
|
||||
"schedule": { "mode": "interval" },
|
||||
"notifications": [
|
||||
{ "type": "telegram", "botToken": "TOKEN", "chatId": "CHAT" }
|
||||
],
|
||||
"maxLinesInMessage": 20,
|
||||
"cooldownMs": 300000
|
||||
},
|
||||
{
|
||||
"id": "disk-space",
|
||||
"name": "Disk Space Warning",
|
||||
"enabled": true,
|
||||
"sources": ["/var/log/syslog"],
|
||||
"patterns": ["No space left", "disk.*9[5-9]%", "filesystem full"],
|
||||
"patternLogic": "any",
|
||||
"timeRange": { "type": "hours", "hours": 2 },
|
||||
"schedule": { "mode": "hourly", "value": "1" },
|
||||
"notifications": [
|
||||
{ "type": "webhook", "url": "https://hooks.slack.com/..." }
|
||||
],
|
||||
"maxLinesInMessage": 5,
|
||||
"cooldownMs": 3600000
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/monitor/rules` | List all rules |
|
||||
| POST | `/api/monitor/rules` | Create a new rule |
|
||||
| PUT | `/api/monitor/rule/:id` | Update a rule |
|
||||
| DELETE | `/api/monitor/rule/:id` | Delete a rule |
|
||||
| POST | `/api/monitor/trigger/:id` | Manually trigger a rule (ignores cooldown) |
|
||||
| GET | `/api/monitor/results` | Get recent alert results |
|
||||
| GET | `/api/monitor/config` | Get monitor configuration |
|
||||
| POST | `/api/monitor/test-notification` | Send a test notification |
|
||||
|
||||
### Automating Rule Creation
|
||||
|
||||
You can fully automate monitor setup via the API:
|
||||
```bash
|
||||
# Create a rule
|
||||
curl -X POST http://localhost:8082/api/monitor/rules \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "My Alert",
|
||||
"enabled": true,
|
||||
"sources": ["/var/log/app.log"],
|
||||
"patterns": ["ERROR", "FATAL"],
|
||||
"patternLogic": "any",
|
||||
"timeRange": {"type": "today"},
|
||||
"notifications": [{"type": "telegram","botToken":"...","chatId":"..."}],
|
||||
"maxLinesInMessage": 20,
|
||||
"cooldownMs": 300000
|
||||
}'
|
||||
|
||||
# Trigger it manually
|
||||
curl -X POST http://localhost:8082/api/monitor/trigger/my-alert
|
||||
|
||||
# Check results
|
||||
curl http://localhost:8082/api/monitor/results
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Alert Message Format
|
||||
|
||||
When a rule triggers, the notification message looks like:
|
||||
|
||||
```
|
||||
🚨 Log Monitor Alert: Root DB Access Detection
|
||||
|
||||
Matches: 6 lines
|
||||
Sources: /var/log/db-audit.log
|
||||
Time: 2026-04-30T07:00:01.234Z
|
||||
Patterns: root.*login | GRANT.*PRIVILEGES | DROP TABLE
|
||||
|
||||
Sample (first 5 of 6):
|
||||
2026-04-30T06:30:01Z WARN [db] root login detected from 10.0.0.1
|
||||
2026-04-30T06:30:05Z WARN [db] root executed: GRANT ALL PRIVILEGES
|
||||
2026-04-30T06:30:06Z ERROR [db] CRITICAL: root executed: DROP TABLE audit_log
|
||||
2026-04-30T07:05:35Z WARN [db] root login detected from 192.168.1.51
|
||||
2026-04-30T07:05:40Z INFO [db] root executed: ALTER TABLE users
|
||||
|
||||
📎 Full results (6 lines) attached as .gz file.
|
||||
```
|
||||
|
||||
If total matches ≤ `maxLinesInMessage`: all lines are included inline.
|
||||
If total matches > `maxLinesInMessage`: first N lines inline + full results as gzipped attachment.
|
||||
Reference in New Issue
Block a user