Initial commit: LogMaster with Viewer Collector Monitor modes

2026-05-04 13:50:15 +02:00
commit 44cd9ab001
231 changed files with 29018 additions and 0 deletions
--- a/docs/COLLECTOR.md
+++ b/docs/COLLECTOR.md
@@ -0,0 +1,358 @@
+# 📡 Log Collector — User Guide
+
+## Overview
+
+The Log Collector receives logs from applications via HTTP and UDP, stores them in named streams, rotates log files based on size/time, and optionally forwards them to other systems (databases, other collectors, files).
+
+---
+
+## Starting the Collector
+
+### Option 1: Environment File
+```bash
+cp .env.collector .env
+node dist-server/server.js
+```
+
+### Option 2: Inline
+```bash
+LOG_MODE=collector PORT=8081 COLLECTOR_UDP_PORT=5140 node dist-server/server.js
+```
+
+### Option 3: Docker
+```bash
+docker run -d -p 8081:8080 -p 5140:5140/udp \
+  -e LOG_MODE=collector \
+  -e COLLECTOR_UDP_PORT=5140 \
+  -e COLLECTOR_ROTATION_COMPRESSION=gzip \
+  -v ./collector-data:/app/collector-data \
+  logviewer
+```
+
+### Option 4: Config File
+```json
+{
+  "mode": "collector",
+  "port": 8081,
+  "collector": {
+    "enabled": true,
+    "storageDir": "./collector-data",
+    "maxLinesPerStream": 100000,
+    "maxStreams": 50,
+    "persistToDisk": true,
+    "udp": { "enabled": true, "port": 5140 },
+    "rotation": {
+      "maxSizeBytes": 5242880,
+      "maxAgeMs": 600000,
+      "compression": "gzip",
+      "fileExtension": ".log"
+    },
+    "forwardTargets": []
+  }
+}
+```
+
+---
+
+## Sending Logs to the Collector
+
+### HTTP POST (Plain Text)
+
+Each line in the body becomes one log entry:
+```bash
+curl -X POST http://localhost:8081/collect/my-app \
+  -d "2026-04-30T10:00:01Z INFO Application started"
+```
+
+Multiple lines:
+```bash
+curl -X POST http://localhost:8081/collect/my-app -d "line 1
+line 2
+line 3"
+```
+
+### HTTP POST (JSON)
+
+Send an array of strings or objects:
+```bash
+curl -X POST http://localhost:8081/collect/my-app/json \
+  -H "Content-Type: application/json" \
+  -d '["INFO App started", "ERROR Crash!", {"ts":"2026-04-30","level":"WARN","msg":"low memory"}]'
+```
+
+### UDP
+
+Format: `streamId|logline` — the part before `|` is the stream name:
+```bash
+echo "my-app|2026-04-30 ERROR database timeout" | nc -u localhost 5140
+```
+
+Without stream ID (auto-named by sender IP):
+```bash
+echo "2026-04-30 INFO hello from UDP" | nc -u localhost 5140
+```
+
+### From Applications
+
+**Node.js:**
+```javascript
+const http = require('http');
+function log(stream, msg) {
+  const req = http.request({ hostname:'localhost', port:8081, path:`/collect/${stream}`, method:'POST' });
+  req.write(`${new Date().toISOString()} ${msg}`);
+  req.end();
+}
+log('my-app', 'INFO Server started on port 3000');
+```
+
+**Python:**
+```python
+import requests, datetime
+def log(stream, msg):
+    ts = datetime.datetime.now().isoformat()
+    requests.post(f'http://localhost:8081/collect/{stream}', data=f'{ts} {msg}')
+
+log('my-app', 'INFO Processing complete')
+```
+
+**Bash (continuous pipe):**
+```bash
+tail -f /var/log/myapp.log | while read line; do
+  curl -s -X POST http://localhost:8081/collect/myapp -d "$line"
+done
+```
+
+### Test Script (included)
+
+Sends random logs every 500ms:
+```bash
+node send-test-logs.js <stream-name> <collector-url>
+node send-test-logs.js demo-app http://localhost:8081
+```
+
+---
+
+## GUI Guide
+
+### Layout
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  📡 Log Collector [ACTIVE]                    Uptime: 5m 32s│
+├──────────────┬──────────────────────────────────────────────┤
+│              │  [🔴 Live Feed] [⚙️ Configuration] [🔀 Fwd]  │
+│  📊 Overview │                                              │
+│  ┌─────────┐ │  Live Feed:                                  │
+│  │ 3       │ │  [demo-app] 10:00:01 INFO App started        │
+│  │ Streams │ │  [demo-app] 10:00:02 ERROR DB timeout  ← red │
+│  ├─────────┤ │  [web-srv] 10:00:03 WARN Slow request ← orange
+│  │ 1.2k    │ │  [demo-app] 10:00:04 INFO Request OK         │
+│  │ Lines   │ │                                              │
+│  ├─────────┤ │  New lines appear with animation             │
+│  │ 120     │ │                                              │
+│  │ /min    │ │                                              │
+│  └─────────┘ │                                              │
+│              │                                              │
+│  📡 Streams  │                                              │
+│  ┌──────────┐│                                              │
+│  │ demo-app ││                                              │
+│  │ 450 lines││                                              │
+│  │ 2s ago   ││                                              │
+│  ├──────────┤│                                              │
+│  │ web-srv  ││                                              │
+│  │ 120 lines││                                              │
+│  └──────────┘│                                              │
+│              │                                              │
+│  🔗 Endpoints                                               │
+│  POST /collect/<id>                                         │
+│  UDP :5140   │                                              │
+└──────────────┴──────────────────────────────────────────────┘
+```
+
+### Tabs
+
+**🔴 Live Feed**
+- Shows incoming log lines in real-time
+- Color-coded by level (red=ERROR, orange=WARN, blue=INFO)
+- New lines animate in from the left
+- Click a stream in the sidebar to filter the feed
+- Pause/Resume button to freeze the feed
+- Clear button to reset
+
+**⚙️ Configuration**
+- Edit all collector settings via the GUI
+- Changes are saved to `logviewer.config.json`
+- Settings: storage dir, max streams, max lines, UDP port, rotation size/age/compression
+
+**🔀 Forwarding**
+- View active forward targets
+- Add new targets (HTTP, File, UDP, Console)
+- Configure batch size and flush interval
+
+---
+
+## Configuration Reference
+
+### Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `LOG_MODE` | `both` | Set to `collector` |
+| `PORT` | `8080` | HTTP port |
+| `COLLECTOR_UDP_ENABLED` | `true` | Enable/disable UDP receiver |
+| `COLLECTOR_UDP_PORT` | `5140` | UDP listen port |
+| `COLLECTOR_STORAGE_DIR` | `./collector-data` | Where to store log files |
+| `COLLECTOR_MAX_LINES` | `100000` | Max lines kept in memory per stream |
+| `COLLECTOR_MAX_STREAMS` | `50` | Max number of streams |
+| `COLLECTOR_PERSIST` | `true` | Write logs to disk |
+| `COLLECTOR_ROTATION_MAX_SIZE` | `5242880` | Rotate after N bytes (default 5MB) |
+| `COLLECTOR_ROTATION_MAX_AGE` | `600000` | Rotate after N ms (default 10min) |
+| `COLLECTOR_ROTATION_COMPRESSION` | `gzip` | `none`, `gzip`, `bzip2`, `xz`, `zstd` |
+| `COLLECTOR_ROTATION_EXTENSION` | `.log` | File extension for active log |
+| `COLLECTOR_FORWARD_TARGETS` | `[]` | JSON array of forward targets |
+
+### Log Rotation
+
+Rotation triggers on whichever condition is met **first**:
+
+| Condition | Variable | Example |
+|-----------|----------|---------|
+| File size exceeds limit | `COLLECTOR_ROTATION_MAX_SIZE=10485760` | Rotate at 10MB |
+| File age exceeds limit | `COLLECTOR_ROTATION_MAX_AGE=3600000` | Rotate every hour |
+
+Rotated files are named with timestamp and compressed:
+```
+collector-data/
+├── my-app.log                              ← current (active)
+├── my-app_2026-04-30T10-00-00-000Z.log.gz  ← rotated + gzipped
+├── my-app_2026-04-29T22-00-00-000Z.log.gz
+└── _streams.json                           ← metadata
+```
+
+### Forward Targets
+
+Forward targets receive copies of all ingested logs. Configure via environment or GUI.
+
+**HTTP Target** — POST logs to another service:
+```json
+{"type":"http","url":"http://elasticsearch:9200/_bulk","format":"json","batchSize":200,"flushIntervalMs":5000}
+```
+
+**File Target** — Append to a file:
+```json
+{"type":"file","path":"/backup/all-logs.txt"}
+```
+
+**UDP Target** — Forward via UDP (e.g., to syslog):
+```json
+{"type":"udp","host":"syslog.internal","port":514}
+```
+
+**Stream Target** — Forward to another stream on the same collector:
+```json
+{"type":"stream","targetStream":"all-logs"}
+```
+This copies all ingested lines from any stream into the `all-logs` stream. Lines are prefixed with `[fwd:source-stream]` to identify the origin. Useful for creating an aggregated "all logs" stream.
+
+**Console Target** — Print to stdout (for debugging):
+```json
+{"type":"console"}
+```
+
+**Multiple targets:**
+```env
+COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://elk:9200/_bulk","format":"json","batchSize":100},{"type":"file","path":"/backup/all.log"},{"type":"console"}]
+```
+
+---
+
+## Example Configurations
+
+### Minimal Collector
+```env
+LOG_MODE=collector
+PORT=8081
+```
+
+### Production Collector (large scale)
+```env
+LOG_MODE=collector
+PORT=8081
+COLLECTOR_UDP_PORT=5140
+COLLECTOR_STORAGE_DIR=/data/logs
+COLLECTOR_MAX_LINES=500000
+COLLECTOR_MAX_STREAMS=200
+COLLECTOR_ROTATION_MAX_SIZE=52428800
+COLLECTOR_ROTATION_MAX_AGE=3600000
+COLLECTOR_ROTATION_COMPRESSION=zstd
+COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://elasticsearch:9200/logs/_bulk","format":"json","batchSize":500,"flushIntervalMs":3000}]
+```
+
+### Collector with multiple forward targets
+```env
+LOG_MODE=collector
+PORT=8081
+COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://loki:3100/loki/api/v1/push","format":"json","batchSize":100},{"type":"file","path":"/archive/all-logs.txt"},{"type":"udp","host":"siem.internal","port":514}]
+```
+
+### Collector with stream aggregation (self-forward)
+```env
+LOG_MODE=collector
+PORT=8081
+COLLECTOR_FORWARD_TARGETS=[{"type":"stream","targetStream":"all-logs"},{"type":"console"}]
+```
+Every log sent to any stream (e.g., `web-app`, `auth-service`) is also copied into the `all-logs` stream. Lines are prefixed with `[fwd:web-app]` etc. so you can still tell where they came from.
+
+### Docker Compose: Collector cluster
+```yaml
+services:
+  collector-1:
+    image: logviewer
+    ports: ["8081:8080", "5140:5140/udp"]
+    environment:
+      LOG_MODE: collector
+      COLLECTOR_UDP_PORT: "5140"
+      COLLECTOR_ROTATION_COMPRESSION: gzip
+      COLLECTOR_FORWARD_TARGETS: '[{"type":"http","url":"http://central:8080/collect/region-eu","format":"json"}]'
+    volumes:
+      - ./data-eu:/app/collector-data
+
+  collector-2:
+    image: logviewer
+    ports: ["8082:8080", "5141:5140/udp"]
+    environment:
+      LOG_MODE: collector
+      COLLECTOR_UDP_PORT: "5140"
+      COLLECTOR_FORWARD_TARGETS: '[{"type":"http","url":"http://central:8080/collect/region-us","format":"json"}]'
+    volumes:
+      - ./data-us:/app/collector-data
+
+  central:
+    image: logviewer
+    ports: ["9090:8080"]
+    environment:
+      LOG_MODE: both
+```
+
+---
+
+## API Reference
+
+| Method | Path | Description |
+|--------|------|-------------|
+| POST | `/collect/:streamId` | Ingest plain text (one line per line) |
+| POST | `/collect/:streamId/json` | Ingest JSON array |
+| GET | `/collector/streams` | List all streams |
+| GET | `/collector/stream/:id?tail=N` | Read stream (optional: last N lines) |
+| GET | `/collector/config` | Current collector config |
+| DELETE | `/collector/stream/:id` | Delete a stream |
+
+### Stream Auto-Creation
+
+Streams are created automatically when you first send logs to them. No pre-configuration needed. Just POST to `/collect/any-name-you-want` and the stream exists.
+
+### Stream Naming
+
+Stream IDs support: `a-z`, `A-Z`, `0-9`, `-`, `_`, `.`
+
+Examples: `my-app`, `web.server.prod`, `auth_service`, `node-1.api`