Keviin/LogMaster
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: LogMaster with Viewer Collector Monitor modes
Some checks failed
LogMaster CI/CD / build-and-test (push) Has been cancelled
Details
LogMaster CI/CD / docker (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Kevin
2026-05-04 13:50:15 +02:00
commit 44cd9ab001
231 changed files with 29018 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

358
docs/COLLECTOR.md Normal file
View File

@@ -0,0 +1,358 @@
# 📡 Log Collector — User Guide
## Overview
The Log Collector receives logs from applications via HTTP and UDP, stores them in named streams, rotates log files based on size/time, and optionally forwards them to other systems (databases, other collectors, files).
---
## Starting the Collector
### Option 1: Environment File
```bash
cp .env.collector .env
node dist-server/server.js
```
### Option 2: Inline
```bash
LOG_MODE=collector PORT=8081 COLLECTOR_UDP_PORT=5140 node dist-server/server.js
```
### Option 3: Docker
```bash
docker run -d -p 8081:8080 -p 5140:5140/udp \
-e LOG_MODE=collector \
-e COLLECTOR_UDP_PORT=5140 \
-e COLLECTOR_ROTATION_COMPRESSION=gzip \
-v ./collector-data:/app/collector-data \
logviewer
```
### Option 4: Config File
```json
{
"mode": "collector",
"port": 8081,
"collector": {
"enabled": true,
"storageDir": "./collector-data",
"maxLinesPerStream": 100000,
"maxStreams": 50,
"persistToDisk": true,
"udp": { "enabled": true, "port": 5140 },
"rotation": {
"maxSizeBytes": 5242880,
"maxAgeMs": 600000,
"compression": "gzip",
"fileExtension": ".log"
},
"forwardTargets": []
}
}
```
---
## Sending Logs to the Collector
### HTTP POST (Plain Text)
Each line in the body becomes one log entry:
```bash
curl -X POST http://localhost:8081/collect/my-app \
-d "2026-04-30T10:00:01Z INFO Application started"
```
Multiple lines:
```bash
curl -X POST http://localhost:8081/collect/my-app -d "line 1
line 2
line 3"
```
### HTTP POST (JSON)
Send an array of strings or objects:
```bash
curl -X POST http://localhost:8081/collect/my-app/json \
-H "Content-Type: application/json" \
-d '["INFO App started", "ERROR Crash!", {"ts":"2026-04-30","level":"WARN","msg":"low memory"}]'
```
### UDP
Format: `streamId|logline` — the part before `|` is the stream name:
```bash
echo "my-app|2026-04-30 ERROR database timeout" | nc -u localhost 5140
```
Without stream ID (auto-named by sender IP):
```bash
echo "2026-04-30 INFO hello from UDP" | nc -u localhost 5140
```
### From Applications
**Node.js:**
```javascript
const http = require('http');
function log(stream, msg) {
const req = http.request({ hostname:'localhost', port:8081, path:`/collect/${stream}`, method:'POST' });
req.write(`${new Date().toISOString()} ${msg}`);
req.end();
}
log('my-app', 'INFO Server started on port 3000');
```
**Python:**
```python
import requests, datetime
def log(stream, msg):
ts = datetime.datetime.now().isoformat()
requests.post(f'http://localhost:8081/collect/{stream}', data=f'{ts} {msg}')
log('my-app', 'INFO Processing complete')
```
**Bash (continuous pipe):**
```bash
tail -f /var/log/myapp.log | while read line; do
curl -s -X POST http://localhost:8081/collect/myapp -d "$line"
done
```
### Test Script (included)
Sends random logs every 500ms:
```bash
node send-test-logs.js <stream-name> <collector-url>
node send-test-logs.js demo-app http://localhost:8081
```
---
## GUI Guide
### Layout
```
┌─────────────────────────────────────────────────────────────┐
│ 📡 Log Collector [ACTIVE] Uptime: 5m 32s│
├──────────────┬──────────────────────────────────────────────┤
│ │ [🔴 Live Feed] [⚙️ Configuration] [🔀 Fwd] │
│ 📊 Overview │ │
│ ┌─────────┐ │ Live Feed: │
│ │ 3 │ │ [demo-app] 10:00:01 INFO App started │
│ │ Streams │ │ [demo-app] 10:00:02 ERROR DB timeout ← red │
│ ├─────────┤ │ [web-srv] 10:00:03 WARN Slow request ← orange
│ │ 1.2k │ │ [demo-app] 10:00:04 INFO Request OK │
│ │ Lines │ │ │
│ ├─────────┤ │ New lines appear with animation │
│ │ 120 │ │ │
│ │ /min │ │ │
│ └─────────┘ │ │
│ │ │
│ 📡 Streams │ │
│ ┌──────────┐│ │
│ │ demo-app ││ │
│ │ 450 lines││ │
│ │ 2s ago ││ │
│ ├──────────┤│ │
│ │ web-srv ││ │
│ │ 120 lines││ │
│ └──────────┘│ │
│ │ │
│ 🔗 Endpoints │
│ POST /collect/<id> │
│ UDP :5140 │ │
└──────────────┴──────────────────────────────────────────────┘
```
### Tabs
**🔴 Live Feed**
- Shows incoming log lines in real-time
- Color-coded by level (red=ERROR, orange=WARN, blue=INFO)
- New lines animate in from the left
- Click a stream in the sidebar to filter the feed
- Pause/Resume button to freeze the feed
- Clear button to reset
**⚙️ Configuration**
- Edit all collector settings via the GUI
- Changes are saved to `logviewer.config.json`
- Settings: storage dir, max streams, max lines, UDP port, rotation size/age/compression
**🔀 Forwarding**
- View active forward targets
- Add new targets (HTTP, File, UDP, Console)
- Configure batch size and flush interval
---
## Configuration Reference
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `LOG_MODE` | `both` | Set to `collector` |
| `PORT` | `8080` | HTTP port |
| `COLLECTOR_UDP_ENABLED` | `true` | Enable/disable UDP receiver |
| `COLLECTOR_UDP_PORT` | `5140` | UDP listen port |
| `COLLECTOR_STORAGE_DIR` | `./collector-data` | Where to store log files |
| `COLLECTOR_MAX_LINES` | `100000` | Max lines kept in memory per stream |
| `COLLECTOR_MAX_STREAMS` | `50` | Max number of streams |
| `COLLECTOR_PERSIST` | `true` | Write logs to disk |
| `COLLECTOR_ROTATION_MAX_SIZE` | `5242880` | Rotate after N bytes (default 5MB) |
| `COLLECTOR_ROTATION_MAX_AGE` | `600000` | Rotate after N ms (default 10min) |
| `COLLECTOR_ROTATION_COMPRESSION` | `gzip` | `none`, `gzip`, `bzip2`, `xz`, `zstd` |
| `COLLECTOR_ROTATION_EXTENSION` | `.log` | File extension for active log |
| `COLLECTOR_FORWARD_TARGETS` | `[]` | JSON array of forward targets |
### Log Rotation
Rotation triggers on whichever condition is met **first**:
| Condition | Variable | Example |
|-----------|----------|---------|
| File size exceeds limit | `COLLECTOR_ROTATION_MAX_SIZE=10485760` | Rotate at 10MB |
| File age exceeds limit | `COLLECTOR_ROTATION_MAX_AGE=3600000` | Rotate every hour |
Rotated files are named with timestamp and compressed:
```
collector-data/
├── my-app.log ← current (active)
├── my-app_2026-04-30T10-00-00-000Z.log.gz ← rotated + gzipped
├── my-app_2026-04-29T22-00-00-000Z.log.gz
└── _streams.json ← metadata
```
### Forward Targets
Forward targets receive copies of all ingested logs. Configure via environment or GUI.
**HTTP Target** — POST logs to another service:
```json
{"type":"http","url":"http://elasticsearch:9200/_bulk","format":"json","batchSize":200,"flushIntervalMs":5000}
```
**File Target** — Append to a file:
```json
{"type":"file","path":"/backup/all-logs.txt"}
```
**UDP Target** — Forward via UDP (e.g., to syslog):
```json
{"type":"udp","host":"syslog.internal","port":514}
```
**Stream Target** — Forward to another stream on the same collector:
```json
{"type":"stream","targetStream":"all-logs"}
```
This copies all ingested lines from any stream into the `all-logs` stream. Lines are prefixed with `[fwd:source-stream]` to identify the origin. Useful for creating an aggregated "all logs" stream.
**Console Target** — Print to stdout (for debugging):
```json
{"type":"console"}
```
**Multiple targets:**
```env
COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://elk:9200/_bulk","format":"json","batchSize":100},{"type":"file","path":"/backup/all.log"},{"type":"console"}]
```
---
## Example Configurations
### Minimal Collector
```env
LOG_MODE=collector
PORT=8081
```
### Production Collector (large scale)
```env
LOG_MODE=collector
PORT=8081
COLLECTOR_UDP_PORT=5140
COLLECTOR_STORAGE_DIR=/data/logs
COLLECTOR_MAX_LINES=500000
COLLECTOR_MAX_STREAMS=200
COLLECTOR_ROTATION_MAX_SIZE=52428800
COLLECTOR_ROTATION_MAX_AGE=3600000
COLLECTOR_ROTATION_COMPRESSION=zstd
COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://elasticsearch:9200/logs/_bulk","format":"json","batchSize":500,"flushIntervalMs":3000}]
```
### Collector with multiple forward targets
```env
LOG_MODE=collector
PORT=8081
COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://loki:3100/loki/api/v1/push","format":"json","batchSize":100},{"type":"file","path":"/archive/all-logs.txt"},{"type":"udp","host":"siem.internal","port":514}]
```
### Collector with stream aggregation (self-forward)
```env
LOG_MODE=collector
PORT=8081
COLLECTOR_FORWARD_TARGETS=[{"type":"stream","targetStream":"all-logs"},{"type":"console"}]
```
Every log sent to any stream (e.g., `web-app`, `auth-service`) is also copied into the `all-logs` stream. Lines are prefixed with `[fwd:web-app]` etc. so you can still tell where they came from.
### Docker Compose: Collector cluster
```yaml
services:
collector-1:
image: logviewer
ports: ["8081:8080", "5140:5140/udp"]
environment:
LOG_MODE: collector
COLLECTOR_UDP_PORT: "5140"
COLLECTOR_ROTATION_COMPRESSION: gzip
COLLECTOR_FORWARD_TARGETS: '[{"type":"http","url":"http://central:8080/collect/region-eu","format":"json"}]'
volumes:
- ./data-eu:/app/collector-data
collector-2:
image: logviewer
ports: ["8082:8080", "5141:5140/udp"]
environment:
LOG_MODE: collector
COLLECTOR_UDP_PORT: "5140"
COLLECTOR_FORWARD_TARGETS: '[{"type":"http","url":"http://central:8080/collect/region-us","format":"json"}]'
volumes:
- ./data-us:/app/collector-data
central:
image: logviewer
ports: ["9090:8080"]
environment:
LOG_MODE: both
```
---
## API Reference
| Method | Path | Description |
|--------|------|-------------|
| POST | `/collect/:streamId` | Ingest plain text (one line per line) |
| POST | `/collect/:streamId/json` | Ingest JSON array |
| GET | `/collector/streams` | List all streams |
| GET | `/collector/stream/:id?tail=N` | Read stream (optional: last N lines) |
| GET | `/collector/config` | Current collector config |
| DELETE | `/collector/stream/:id` | Delete a stream |
### Stream Auto-Creation
Streams are created automatically when you first send logs to them. No pre-configuration needed. Just POST to `/collect/any-name-you-want` and the stream exists.
### Stream Naming
Stream IDs support: `a-z`, `A-Z`, `0-9`, `-`, `_`, `.`
Examples: `my-app`, `web.server.prod`, `auth_service`, `node-1.api`

377
docs/MONITOR.md Normal file
View File

@@ -0,0 +1,377 @@
# 🔍 Log Monitor — User Guide
## Overview
The Log Monitor scans log files on a schedule, matches regex patterns, and sends alerts via Telegram, Webhooks, Microsoft Teams, or Email. It supports time-based filtering, scheduled scans (daily, hourly, continuous), and automatic attachment of full results when matches exceed a threshold.
---
## Starting the Monitor
### Option 1: Environment File
```bash
cp .env.monitor .env
node dist-server/server.js
```
### Option 2: Inline
```bash
LOG_MODE=monitor PORT=8082 MONITOR_SCAN_INTERVAL=30000 node dist-server/server.js
```
### Option 3: Docker
```bash
docker run -d -p 8082:8080 \
-e LOG_MODE=monitor \
-e MONITOR_SCAN_INTERVAL=30000 \
-v /var/log:/logs:ro \
logviewer
```
### Option 4: Config File
```json
{
"mode": "monitor",
"port": 8082,
"monitor": {
"enabled": true,
"scanIntervalMs": 30000,
"rules": []
}
}
```
---
## How It Works
```
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
│ Log Files │────▶│ Scan Engine │────▶│ Pattern Match │
│ (sources) │ │ (interval) │ │ (regex) │
└─────────────┘ └──────────────┘ └────────┬────────┘
│
matches found?
│
┌──────────────┴──────────────┐
│ │
≤ maxLines > maxLines
│ │
┌─────▼─────┐ ┌───────▼───────┐
│ Inline │ │ Inline (N) │
│ in alert │ │ + .gz attach │
└─────┬─────┘ └───────┬───────┘
│ │
└──────────────┬──────────────┘
│
┌──────────────▼──────────────┐
│ Send Notifications │
│ Telegram │ Webhook │ Teams │
└─────────────────────────────┘
```
1. **Scan** — Every N seconds (configurable), the monitor reads all source files
2. **Filter** — Only files matching the time range are scanned (today, this week, etc.)
3. **Match** — Each line is tested against the configured regex patterns
4. **Alert** — If matches are found AND cooldown has expired, notifications are sent
5. **Attach** — If matches exceed `maxLinesInMessage`, full results are gzipped and attached
---
## GUI Guide
### Layout
```
┌─────────────────────────────────────────────────────────────┐
│ 🔍 Log Monitor 2 active rules · 3 alerts│
├──────────────┬──────────────────────────────────────────────┤
│ │ │
│ 📋 Rules │ Rule Editor / Alert Detail │
│ ┌──────────┐│ │
│ │●Root DB ││ [General] │
│ │ 2 src ││ Name: Root DB Access Detection │
│ │ 6 match ││ Enabled: Yes │
│ ├──────────┤│ │
│ │●Failed ││ [Sources] │
│ │ 1 src ││ /var/log/db-audit.log │
│ │ 4 match ││ /var/log/db-audit-*.log.gz │
│ └──────────┘│ │
│ │ [Patterns] │
│ 📊 Alerts │ root.*login │
│ ┌──────────┐│ GRANT.*PRIVILEGES │
│ │🚨 Root DB││ DROP TABLE │
│ │ 6 matches││ │
│ │ 10:30:01 ││ [Schedule] │
│ ├──────────┤│ Mode: Daily at 07:00 │
│ │🚨 Failed ││ │
│ │ 4 matches││ [Notifications] │
│ │ 10:30:01 ││ [TELEGRAM] Bot: 811... Chat: 859... │
│ └──────────┘│ [WEBHOOK] https://hooks.slack.com/... │
│ │ │
│ │ [💾 Save] [🧪 Test Now] [🗑 Delete] │
└──────────────┴──────────────────────────────────────────────┘
```
### Creating a Rule
1. Click **"+ New"** in the sidebar
2. Fill in the fields:
- **Name** — descriptive name for the rule
- **Sources** — one file path per line (supports globs like `/var/log/*.log`)
- **Time Range** — which files to scan (today, week, month, last N hours, all)
- **Patterns** — one regex per line
- **Match Logic** — "any" (OR) or "all" (AND)
- **Schedule** — continuous, daily at time, or every N hours
- **Max Lines** — how many lines to include inline in notifications
- **Cooldown** — minimum time between alerts (prevents spam)
3. Add notification channels
4. Click **"💾 Save Rule"**
### Testing a Rule
1. Select a rule from the sidebar
2. Click **"🧪 Test Now"**
3. The rule runs immediately and shows results below the form
4. This does NOT respect cooldown — always runs
### Testing Notifications
1. In the Notifications section, configure a channel
2. Click **"🧪 Test Notification"**
3. A test message is sent to verify the channel works
### Viewing Alerts
- Alerts appear in the **"📊 Recent Alerts"** section of the sidebar
- Click an alert to see the full detail view with all matched lines
- Lines are color-coded: red for ERROR/FATAL, orange for WARN
---
## Configuration Reference
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `LOG_MODE` | `both` | Set to `monitor` |
| `PORT` | `8080` | HTTP port |
| `MONITOR_ENABLED` | `true` | Enable/disable scanning |
| `MONITOR_SCAN_INTERVAL` | `60000` | Scan interval in ms |
| `MONITOR_STORAGE_DIR` | `./monitor-data` | Where to store state |
| `MONITOR_RULES` | `[]` | JSON array of rules (see below) |
### Rule Structure
```json
{
"id": "unique-id",
"name": "Human-readable name",
"enabled": true,
"sources": ["/path/to/logs/*.log"],
"patterns": ["regex1", "regex2"],
"patternLogic": "any",
"timeRange": { "type": "week", "hours": 24 },
"schedule": { "mode": "daily", "value": "07:00", "timezone": "Europe/Berlin" },
"notifications": [
{ "type": "telegram", "botToken": "123:ABC", "chatId": "-100123" }
],
"maxLinesInMessage": 20,
"cooldownMs": 300000,
"matchCount": 0,
"lastTriggered": null
}
```
### Time Range Options
| Type | Description | Use Case |
|------|-------------|----------|
| `today` | Only files modified today | Daily reports |
| `week` | Files from last 7 days | Weekly audits |
| `month` | Files from last 30 days | Monthly reviews |
| `hours` | Files from last N hours | Recent activity |
| `all` | All files regardless of age | Full history scan |
### Schedule Options
| Mode | Value | Description |
|------|-------|-------------|
| `interval` | — | Runs every scan interval (continuous monitoring) |
| `daily` | `07:00` | Runs once per day at the specified time |
| `hourly` | `2` | Runs every N hours |
### Notification Channels
**Telegram:**
```json
{ "type": "telegram", "botToken": "123456:ABC-DEF...", "chatId": "8599028500" }
```
**Webhook (Slack, Discord, custom):**
```json
{ "type": "webhook", "url": "https://hooks.slack.com/services/T.../B.../..." }
```
**Microsoft Teams:**
```json
{ "type": "teams", "url": "https://outlook.office.com/webhook/..." }
```
**Email (SMTP):**
```json
{ "type": "email", "to": "admin@company.com", "smtpHost": "smtp.gmail.com", "smtpPort": 587, "smtpUser": "alerts@company.com", "smtpPass": "app-password" }
```
---
## Example Configurations
### 1. Real-time Root Login Detection
Scans every 10 seconds, alerts immediately via Telegram:
```env
LOG_MODE=monitor
PORT=8082
MONITOR_SCAN_INTERVAL=10000
MONITOR_RULES=[{"id":"root-login","name":"Root Login Alert","enabled":true,"sources":["/var/log/auth.log"],"patterns":["session opened for user root","su.*root","sudo.*root"],"patternLogic":"any","timeRange":{"type":"today"},"schedule":{"mode":"interval"},"notifications":[{"type":"telegram","botToken":"YOUR_BOT_TOKEN","chatId":"YOUR_CHAT_ID"}],"maxLinesInMessage":10,"cooldownMs":60000,"matchCount":0,"lastTriggered":null}]
```
### 2. Daily DB Audit Report at 7:00 AM
Scans once per day, sends full report:
```env
LOG_MODE=monitor
PORT=8082
MONITOR_SCAN_INTERVAL=60000
MONITOR_RULES=[{"id":"db-audit","name":"Daily DB Audit","enabled":true,"sources":["/var/log/postgresql/audit.log","/var/log/mysql/audit*.log"],"patterns":["admin.*LOGIN","GRANT","DROP","ALTER.*TABLE","CREATE USER"],"patternLogic":"any","timeRange":{"type":"today"},"schedule":{"mode":"daily","value":"07:00","timezone":"Europe/Berlin"},"notifications":[{"type":"telegram","botToken":"TOKEN","chatId":"CHAT_ID"},{"type":"webhook","url":"https://hooks.slack.com/services/..."}],"maxLinesInMessage":20,"cooldownMs":86400000,"matchCount":0,"lastTriggered":null}]
```
### 3. Error Spike Detection (every 5 minutes)
```env
LOG_MODE=monitor
PORT=8082
MONITOR_SCAN_INTERVAL=30000
MONITOR_RULES=[{"id":"error-spike","name":"Error Spike","enabled":true,"sources":["/app/logs/production.log"],"patterns":["\\bERROR\\b","\\bFATAL\\b","Exception","panic:"],"patternLogic":"any","timeRange":{"type":"hours","hours":1},"schedule":{"mode":"hourly","value":"0.083"},"notifications":[{"type":"teams","url":"https://outlook.office.com/webhook/..."}],"maxLinesInMessage":30,"cooldownMs":300000,"matchCount":0,"lastTriggered":null}]
```
### 4. Security: Unauthorized Access Attempts
```env
MONITOR_RULES=[{"id":"unauth","name":"Unauthorized Access","enabled":true,"sources":["/var/log/nginx/access.log","/var/log/nginx/error.log"],"patterns":["403 Forbidden","401 Unauthorized","\\.\\./\\.\\./","etc/passwd","<script>"],"patternLogic":"any","timeRange":{"type":"today"},"schedule":{"mode":"interval"},"notifications":[{"type":"telegram","botToken":"TOKEN","chatId":"CHAT"},{"type":"webhook","url":"https://siem.internal/alerts"}],"maxLinesInMessage":15,"cooldownMs":120000,"matchCount":0,"lastTriggered":null}]
```
### 5. Multiple Rules via Config File
```json
{
"mode": "monitor",
"port": 8082,
"monitor": {
"enabled": true,
"scanIntervalMs": 30000,
"rules": [
{
"id": "root-access",
"name": "Root DB Access",
"enabled": true,
"sources": ["/var/log/db-audit.log"],
"patterns": ["root.*login", "GRANT", "DROP TABLE"],
"patternLogic": "any",
"timeRange": { "type": "today" },
"schedule": { "mode": "interval" },
"notifications": [
{ "type": "telegram", "botToken": "TOKEN", "chatId": "CHAT" }
],
"maxLinesInMessage": 20,
"cooldownMs": 300000
},
{
"id": "disk-space",
"name": "Disk Space Warning",
"enabled": true,
"sources": ["/var/log/syslog"],
"patterns": ["No space left", "disk.*9[5-9]%", "filesystem full"],
"patternLogic": "any",
"timeRange": { "type": "hours", "hours": 2 },
"schedule": { "mode": "hourly", "value": "1" },
"notifications": [
{ "type": "webhook", "url": "https://hooks.slack.com/..." }
],
"maxLinesInMessage": 5,
"cooldownMs": 3600000
}
]
}
}
```
---
## API Reference
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/monitor/rules` | List all rules |
| POST | `/api/monitor/rules` | Create a new rule |
| PUT | `/api/monitor/rule/:id` | Update a rule |
| DELETE | `/api/monitor/rule/:id` | Delete a rule |
| POST | `/api/monitor/trigger/:id` | Manually trigger a rule (ignores cooldown) |
| GET | `/api/monitor/results` | Get recent alert results |
| GET | `/api/monitor/config` | Get monitor configuration |
| POST | `/api/monitor/test-notification` | Send a test notification |
### Automating Rule Creation
You can fully automate monitor setup via the API:
```bash
# Create a rule
curl -X POST http://localhost:8082/api/monitor/rules \
-H "Content-Type: application/json" \
-d '{
"name": "My Alert",
"enabled": true,
"sources": ["/var/log/app.log"],
"patterns": ["ERROR", "FATAL"],
"patternLogic": "any",
"timeRange": {"type": "today"},
"notifications": [{"type": "telegram","botToken":"...","chatId":"..."}],
"maxLinesInMessage": 20,
"cooldownMs": 300000
}'
# Trigger it manually
curl -X POST http://localhost:8082/api/monitor/trigger/my-alert
# Check results
curl http://localhost:8082/api/monitor/results
```
---
## Alert Message Format
When a rule triggers, the notification message looks like:
```
🚨 Log Monitor Alert: Root DB Access Detection
Matches: 6 lines
Sources: /var/log/db-audit.log
Time: 2026-04-30T07:00:01.234Z
Patterns: root.*login | GRANT.*PRIVILEGES | DROP TABLE
Sample (first 5 of 6):
2026-04-30T06:30:01Z WARN [db] root login detected from 10.0.0.1
2026-04-30T06:30:05Z WARN [db] root executed: GRANT ALL PRIVILEGES
2026-04-30T06:30:06Z ERROR [db] CRITICAL: root executed: DROP TABLE audit_log
2026-04-30T07:05:35Z WARN [db] root login detected from 192.168.1.51
2026-04-30T07:05:40Z INFO [db] root executed: ALTER TABLE users
📎 Full results (6 lines) attached as .gz file.
```
If total matches ≤ `maxLinesInMessage`: all lines are included inline.
If total matches > `maxLinesInMessage`: first N lines inline + full results as gzipped attachment.

229
docs/VIEWER.md Normal file
View File

@@ -0,0 +1,229 @@
# 📋 Log Viewer — User Guide
## Overview
The Log Viewer is a web-based interface for reading, filtering, searching, and exporting log files from multiple sources including local files, S3 buckets, and remote collector streams.
---
## Starting the Viewer
### Option 1: Environment File
```bash
cp .env.viewer .env
node dist-server/server.js
```
### Option 2: Inline Environment Variables
```bash
LOG_MODE=viewer PORT=9090 LOG_SOURCES=/var/log/syslog,/app/logs/*.log node dist-server/server.js
```
### Option 3: Docker
```bash
docker run -d -p 9090:8080 \
-e LOG_MODE=viewer \
-e LOG_SOURCES=/logs/*.log \
-v /var/log:/logs:ro \
logviewer
```
### Option 4: Config File (`logviewer.config.json`)
```json
{
"mode": "viewer",
"port": 9090,
"viewer": {
"sources": ["/var/log/syslog", "/app/logs/*.log"],
"theme": "dark",
"refreshInterval": 2000,
"tailMode": true,
"maxLines": 50000
},
"remoteCollector": {
"url": "http://collector-server:8081"
}
}
```
---
## GUI Guide
### Layout
```
┌─────────────────────────────────────────────────────────────┐
│ [📂 File] [☁️ S3] [🔄 Reload] | [Search...] | [Levels ▼] │ ← Toolbar
├──────────┬──────────────────────────────────────────────────┤
│ │ │
│ Sources │ Log Output │
│ Sidebar │ │
│ │ Each line shows: │
│ 📁 Files│ [LineNo] [Source] [Timestamp] [LEVEL] Content │
│ ☁️ S3 │ │
│ 📡 Local│ New lines flash blue briefly │
│ 🌐 Remote │
│ │ │
├──────────┴──────────────────────────────────────────────────┤
│ Status: Ready · 1247 lines · 1247 shown TAIL WRAP │ ← Status Bar
└─────────────────────────────────────────────────────────────┘
```
### Toolbar Actions
| Button | Action |
|--------|--------|
| 📂 File | Add a local file path (supports globs like `/var/log/*.log`) |
| ☁️ S3 | Add an S3 bucket source (access key, secret, region, bucket, prefix) |
| 🔄 Reload | Refresh all sources |
| ⬇️ Tail | Toggle auto-scroll to new lines |
| ↩️ Wrap | Toggle word wrap for long lines |
| 🕐 Time | Show/hide timestamp column |
| Search box | Type to filter. Click `.*` for regex, `Aa` for case-sensitive |
| Level dropdown | Filter by log level (ERROR, WARN, INFO, etc.) |
| Source dropdown | Filter by specific source file |
| 💾 Export | Download visible logs as TXT or JSON |
| 🌙/☀️ | Toggle dark/light theme |
### Sidebar
The sidebar shows all configured sources grouped by type:
- **📁 Files** — Local log files
- **☁️ S3** — S3 bucket sources
- **📡 Collector Streams** — Local collector streams (if mode=both)
- **🌐 Remote Streams** — Streams from a remote collector
Click a source to view only its logs. Click again or select "All Sources" to see everything.
### Live Highlighting
When tail mode is active, new log lines that appear during auto-refresh are briefly highlighted with a blue left border and background fade. This makes it easy to spot new entries in a busy log.
### Context Menu (Right-Click)
Right-click any log line for:
- 📋 Copy Line
- 🔖 Toggle Bookmark
- 🔍 Filter by this level
- 📋 Copy all visible lines
### Keyboard Shortcuts
| Key | Action |
|-----|--------|
| `Ctrl+F` | Focus search box |
| `Ctrl+Shift+F` | Enable regex mode + focus search |
| `Ctrl+T` | Toggle tail mode |
| `Ctrl+W` | Toggle word wrap |
| `Ctrl+G` | Go to line number |
| `Ctrl+Shift+T` | Toggle theme |
| `Ctrl+=` / `Ctrl+-` | Zoom in/out |
| `Ctrl+0` | Reset zoom |
| `Home` / `End` | Scroll to top/bottom |
| `Escape` | Clear search / close modals |
---
## Configuration Reference
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `LOG_MODE` | `both` | Set to `viewer` for viewer-only mode |
| `PORT` | `8080` | HTTP port |
| `LOG_SOURCES` | — | Comma-separated file paths or globs |
| `LOG_THEME` | `dark` | `dark` or `light` |
| `LOG_REFRESH_INTERVAL` | `2000` | Auto-refresh interval in ms |
| `LOG_TAIL_MODE` | `true` | Start with tail mode enabled |
| `LOG_MAX_LINES` | `50000` | Max lines loaded per file |
| `LOG_DEFAULT_LEVEL` | `ALL` | Default level filter |
| `REMOTE_COLLECTOR_URL` | — | URL of a remote collector to read streams from |
| `S3_ACCESS_KEY_ID` | — | Global S3 access key |
| `S3_SECRET_ACCESS_KEY` | — | Global S3 secret key |
| `S3_REGION` | `us-east-1` | Global S3 region |
| `S3_ENDPOINT` | — | Custom endpoint for S3-compatible services |
| `S3_SOURCE_<NAME>` | — | S3 source in format `bucket:prefix` |
### Supported File Formats
| Format | Extensions | How it works |
|--------|-----------|--------------|
| Plain text | `.log`, `.txt`, no extension | Direct read |
| Gzip | `.gz` | Decompressed with Node.js zlib |
| Bzip2 | `.bz2` | Decompressed via `bzcat` CLI |
| XZ | `.xz` | Decompressed via `xzcat` CLI |
| Zstandard | `.zst` | Decompressed via `zstdcat` CLI |
---
## Example Configurations
### Basic: View local syslog
```env
LOG_MODE=viewer
PORT=9090
LOG_SOURCES=/var/log/syslog
```
### Multiple sources with glob
```env
LOG_MODE=viewer
PORT=9090
LOG_SOURCES=/var/log/syslog,/var/log/auth.log,/app/logs/*.log,/app/logs/archived/*.gz
```
### Viewer connected to remote collector
```env
LOG_MODE=viewer
PORT=9090
REMOTE_COLLECTOR_URL=http://10.0.1.50:8081
LOG_SOURCES=/var/log/local-app.log
```
### Viewer with S3 sources
```env
LOG_MODE=viewer
PORT=9090
S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
S3_REGION=eu-central-1
S3_SOURCE_PROD=company-logs:production/app/
S3_SOURCE_INFRA=company-logs:infrastructure/
```
### Viewer with MinIO (S3-compatible)
```env
LOG_MODE=viewer
PORT=9090
S3_ACCESS_KEY_ID=minioadmin
S3_SECRET_ACCESS_KEY=minioadmin
S3_REGION=us-east-1
S3_ENDPOINT=http://minio.local:9000
S3_SOURCE_LOGS=my-bucket:logs/
```
### Docker Compose: Viewer + Collector
```yaml
services:
collector:
image: logviewer
ports: ["8081:8080", "5140:5140/udp"]
environment:
LOG_MODE: collector
COLLECTOR_UDP_PORT: "5140"
volumes:
- collector-data:/app/collector-data
viewer:
image: logviewer
ports: ["9090:8080"]
environment:
LOG_MODE: viewer
REMOTE_COLLECTOR_URL: http://collector:8080
depends_on: [collector]
volumes:
collector-data:
```