359 lines
12 KiB
Markdown
359 lines
12 KiB
Markdown
# 📡 Log Collector — User Guide
|
|
|
|
## Overview
|
|
|
|
The Log Collector receives logs from applications via HTTP and UDP, stores them in named streams, rotates log files based on size/time, and optionally forwards them to other systems (databases, other collectors, files).
|
|
|
|
---
|
|
|
|
## Starting the Collector
|
|
|
|
### Option 1: Environment File
|
|
```bash
|
|
cp .env.collector .env
|
|
node dist-server/server.js
|
|
```
|
|
|
|
### Option 2: Inline
|
|
```bash
|
|
LOG_MODE=collector PORT=8081 COLLECTOR_UDP_PORT=5140 node dist-server/server.js
|
|
```
|
|
|
|
### Option 3: Docker
|
|
```bash
|
|
docker run -d -p 8081:8080 -p 5140:5140/udp \
|
|
-e LOG_MODE=collector \
|
|
-e COLLECTOR_UDP_PORT=5140 \
|
|
-e COLLECTOR_ROTATION_COMPRESSION=gzip \
|
|
-v ./collector-data:/app/collector-data \
|
|
logviewer
|
|
```
|
|
|
|
### Option 4: Config File
|
|
```json
|
|
{
|
|
"mode": "collector",
|
|
"port": 8081,
|
|
"collector": {
|
|
"enabled": true,
|
|
"storageDir": "./collector-data",
|
|
"maxLinesPerStream": 100000,
|
|
"maxStreams": 50,
|
|
"persistToDisk": true,
|
|
"udp": { "enabled": true, "port": 5140 },
|
|
"rotation": {
|
|
"maxSizeBytes": 5242880,
|
|
"maxAgeMs": 600000,
|
|
"compression": "gzip",
|
|
"fileExtension": ".log"
|
|
},
|
|
"forwardTargets": []
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Sending Logs to the Collector
|
|
|
|
### HTTP POST (Plain Text)
|
|
|
|
Each line in the body becomes one log entry:
|
|
```bash
|
|
curl -X POST http://localhost:8081/collect/my-app \
|
|
-d "2026-04-30T10:00:01Z INFO Application started"
|
|
```
|
|
|
|
Multiple lines:
|
|
```bash
|
|
curl -X POST http://localhost:8081/collect/my-app -d "line 1
|
|
line 2
|
|
line 3"
|
|
```
|
|
|
|
### HTTP POST (JSON)
|
|
|
|
Send an array of strings or objects:
|
|
```bash
|
|
curl -X POST http://localhost:8081/collect/my-app/json \
|
|
-H "Content-Type: application/json" \
|
|
-d '["INFO App started", "ERROR Crash!", {"ts":"2026-04-30","level":"WARN","msg":"low memory"}]'
|
|
```
|
|
|
|
### UDP
|
|
|
|
Format: `streamId|logline` — the part before `|` is the stream name:
|
|
```bash
|
|
echo "my-app|2026-04-30 ERROR database timeout" | nc -u localhost 5140
|
|
```
|
|
|
|
Without stream ID (auto-named by sender IP):
|
|
```bash
|
|
echo "2026-04-30 INFO hello from UDP" | nc -u localhost 5140
|
|
```
|
|
|
|
### From Applications
|
|
|
|
**Node.js:**
|
|
```javascript
|
|
const http = require('http');
|
|
function log(stream, msg) {
|
|
const req = http.request({ hostname:'localhost', port:8081, path:`/collect/${stream}`, method:'POST' });
|
|
req.write(`${new Date().toISOString()} ${msg}`);
|
|
req.end();
|
|
}
|
|
log('my-app', 'INFO Server started on port 3000');
|
|
```
|
|
|
|
**Python:**
|
|
```python
|
|
import requests, datetime
|
|
def log(stream, msg):
|
|
ts = datetime.datetime.now().isoformat()
|
|
requests.post(f'http://localhost:8081/collect/{stream}', data=f'{ts} {msg}')
|
|
|
|
log('my-app', 'INFO Processing complete')
|
|
```
|
|
|
|
**Bash (continuous pipe):**
|
|
```bash
|
|
tail -f /var/log/myapp.log | while read line; do
|
|
curl -s -X POST http://localhost:8081/collect/myapp -d "$line"
|
|
done
|
|
```
|
|
|
|
### Test Script (included)
|
|
|
|
Sends random logs every 500ms:
|
|
```bash
|
|
node send-test-logs.js <stream-name> <collector-url>
|
|
node send-test-logs.js demo-app http://localhost:8081
|
|
```
|
|
|
|
---
|
|
|
|
## GUI Guide
|
|
|
|
### Layout
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ 📡 Log Collector [ACTIVE] Uptime: 5m 32s│
|
|
├──────────────┬──────────────────────────────────────────────┤
|
|
│ │ [🔴 Live Feed] [⚙️ Configuration] [🔀 Fwd] │
|
|
│ 📊 Overview │ │
|
|
│ ┌─────────┐ │ Live Feed: │
|
|
│ │ 3 │ │ [demo-app] 10:00:01 INFO App started │
|
|
│ │ Streams │ │ [demo-app] 10:00:02 ERROR DB timeout ← red │
|
|
│ ├─────────┤ │ [web-srv] 10:00:03 WARN Slow request ← orange
|
|
│ │ 1.2k │ │ [demo-app] 10:00:04 INFO Request OK │
|
|
│ │ Lines │ │ │
|
|
│ ├─────────┤ │ New lines appear with animation │
|
|
│ │ 120 │ │ │
|
|
│ │ /min │ │ │
|
|
│ └─────────┘ │ │
|
|
│ │ │
|
|
│ 📡 Streams │ │
|
|
│ ┌──────────┐│ │
|
|
│ │ demo-app ││ │
|
|
│ │ 450 lines││ │
|
|
│ │ 2s ago ││ │
|
|
│ ├──────────┤│ │
|
|
│ │ web-srv ││ │
|
|
│ │ 120 lines││ │
|
|
│ └──────────┘│ │
|
|
│ │ │
|
|
│ 🔗 Endpoints │
|
|
│ POST /collect/<id> │
|
|
│ UDP :5140 │ │
|
|
└──────────────┴──────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Tabs
|
|
|
|
**🔴 Live Feed**
|
|
- Shows incoming log lines in real-time
|
|
- Color-coded by level (red=ERROR, orange=WARN, blue=INFO)
|
|
- New lines animate in from the left
|
|
- Click a stream in the sidebar to filter the feed
|
|
- Pause/Resume button to freeze the feed
|
|
- Clear button to reset
|
|
|
|
**⚙️ Configuration**
|
|
- Edit all collector settings via the GUI
|
|
- Changes are saved to `logviewer.config.json`
|
|
- Settings: storage dir, max streams, max lines, UDP port, rotation size/age/compression
|
|
|
|
**🔀 Forwarding**
|
|
- View active forward targets
|
|
- Add new targets (HTTP, File, UDP, Console)
|
|
- Configure batch size and flush interval
|
|
|
|
---
|
|
|
|
## Configuration Reference
|
|
|
|
### Environment Variables
|
|
|
|
| Variable | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `LOG_MODE` | `both` | Set to `collector` |
|
|
| `PORT` | `8080` | HTTP port |
|
|
| `COLLECTOR_UDP_ENABLED` | `true` | Enable/disable UDP receiver |
|
|
| `COLLECTOR_UDP_PORT` | `5140` | UDP listen port |
|
|
| `COLLECTOR_STORAGE_DIR` | `./collector-data` | Where to store log files |
|
|
| `COLLECTOR_MAX_LINES` | `100000` | Max lines kept in memory per stream |
|
|
| `COLLECTOR_MAX_STREAMS` | `50` | Max number of streams |
|
|
| `COLLECTOR_PERSIST` | `true` | Write logs to disk |
|
|
| `COLLECTOR_ROTATION_MAX_SIZE` | `5242880` | Rotate after N bytes (default 5MB) |
|
|
| `COLLECTOR_ROTATION_MAX_AGE` | `600000` | Rotate after N ms (default 10min) |
|
|
| `COLLECTOR_ROTATION_COMPRESSION` | `gzip` | `none`, `gzip`, `bzip2`, `xz`, `zstd` |
|
|
| `COLLECTOR_ROTATION_EXTENSION` | `.log` | File extension for active log |
|
|
| `COLLECTOR_FORWARD_TARGETS` | `[]` | JSON array of forward targets |
|
|
|
|
### Log Rotation
|
|
|
|
Rotation triggers on whichever condition is met **first**:
|
|
|
|
| Condition | Variable | Example |
|
|
|-----------|----------|---------|
|
|
| File size exceeds limit | `COLLECTOR_ROTATION_MAX_SIZE=10485760` | Rotate at 10MB |
|
|
| File age exceeds limit | `COLLECTOR_ROTATION_MAX_AGE=3600000` | Rotate every hour |
|
|
|
|
Rotated files are named with timestamp and compressed:
|
|
```
|
|
collector-data/
|
|
├── my-app.log ← current (active)
|
|
├── my-app_2026-04-30T10-00-00-000Z.log.gz ← rotated + gzipped
|
|
├── my-app_2026-04-29T22-00-00-000Z.log.gz
|
|
└── _streams.json ← metadata
|
|
```
|
|
|
|
### Forward Targets
|
|
|
|
Forward targets receive copies of all ingested logs. Configure via environment or GUI.
|
|
|
|
**HTTP Target** — POST logs to another service:
|
|
```json
|
|
{"type":"http","url":"http://elasticsearch:9200/_bulk","format":"json","batchSize":200,"flushIntervalMs":5000}
|
|
```
|
|
|
|
**File Target** — Append to a file:
|
|
```json
|
|
{"type":"file","path":"/backup/all-logs.txt"}
|
|
```
|
|
|
|
**UDP Target** — Forward via UDP (e.g., to syslog):
|
|
```json
|
|
{"type":"udp","host":"syslog.internal","port":514}
|
|
```
|
|
|
|
**Stream Target** — Forward to another stream on the same collector:
|
|
```json
|
|
{"type":"stream","targetStream":"all-logs"}
|
|
```
|
|
This copies all ingested lines from any stream into the `all-logs` stream. Lines are prefixed with `[fwd:source-stream]` to identify the origin. Useful for creating an aggregated "all logs" stream.
|
|
|
|
**Console Target** — Print to stdout (for debugging):
|
|
```json
|
|
{"type":"console"}
|
|
```
|
|
|
|
**Multiple targets:**
|
|
```env
|
|
COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://elk:9200/_bulk","format":"json","batchSize":100},{"type":"file","path":"/backup/all.log"},{"type":"console"}]
|
|
```
|
|
|
|
---
|
|
|
|
## Example Configurations
|
|
|
|
### Minimal Collector
|
|
```env
|
|
LOG_MODE=collector
|
|
PORT=8081
|
|
```
|
|
|
|
### Production Collector (large scale)
|
|
```env
|
|
LOG_MODE=collector
|
|
PORT=8081
|
|
COLLECTOR_UDP_PORT=5140
|
|
COLLECTOR_STORAGE_DIR=/data/logs
|
|
COLLECTOR_MAX_LINES=500000
|
|
COLLECTOR_MAX_STREAMS=200
|
|
COLLECTOR_ROTATION_MAX_SIZE=52428800
|
|
COLLECTOR_ROTATION_MAX_AGE=3600000
|
|
COLLECTOR_ROTATION_COMPRESSION=zstd
|
|
COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://elasticsearch:9200/logs/_bulk","format":"json","batchSize":500,"flushIntervalMs":3000}]
|
|
```
|
|
|
|
### Collector with multiple forward targets
|
|
```env
|
|
LOG_MODE=collector
|
|
PORT=8081
|
|
COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://loki:3100/loki/api/v1/push","format":"json","batchSize":100},{"type":"file","path":"/archive/all-logs.txt"},{"type":"udp","host":"siem.internal","port":514}]
|
|
```
|
|
|
|
### Collector with stream aggregation (self-forward)
|
|
```env
|
|
LOG_MODE=collector
|
|
PORT=8081
|
|
COLLECTOR_FORWARD_TARGETS=[{"type":"stream","targetStream":"all-logs"},{"type":"console"}]
|
|
```
|
|
Every log sent to any stream (e.g., `web-app`, `auth-service`) is also copied into the `all-logs` stream. Lines are prefixed with `[fwd:web-app]` etc. so you can still tell where they came from.
|
|
|
|
### Docker Compose: Collector cluster
|
|
```yaml
|
|
services:
|
|
collector-1:
|
|
image: logviewer
|
|
ports: ["8081:8080", "5140:5140/udp"]
|
|
environment:
|
|
LOG_MODE: collector
|
|
COLLECTOR_UDP_PORT: "5140"
|
|
COLLECTOR_ROTATION_COMPRESSION: gzip
|
|
COLLECTOR_FORWARD_TARGETS: '[{"type":"http","url":"http://central:8080/collect/region-eu","format":"json"}]'
|
|
volumes:
|
|
- ./data-eu:/app/collector-data
|
|
|
|
collector-2:
|
|
image: logviewer
|
|
ports: ["8082:8080", "5141:5140/udp"]
|
|
environment:
|
|
LOG_MODE: collector
|
|
COLLECTOR_UDP_PORT: "5140"
|
|
COLLECTOR_FORWARD_TARGETS: '[{"type":"http","url":"http://central:8080/collect/region-us","format":"json"}]'
|
|
volumes:
|
|
- ./data-us:/app/collector-data
|
|
|
|
central:
|
|
image: logviewer
|
|
ports: ["9090:8080"]
|
|
environment:
|
|
LOG_MODE: both
|
|
```
|
|
|
|
---
|
|
|
|
## API Reference
|
|
|
|
| Method | Path | Description |
|
|
|--------|------|-------------|
|
|
| POST | `/collect/:streamId` | Ingest plain text (one line per line) |
|
|
| POST | `/collect/:streamId/json` | Ingest JSON array |
|
|
| GET | `/collector/streams` | List all streams |
|
|
| GET | `/collector/stream/:id?tail=N` | Read stream (optional: last N lines) |
|
|
| GET | `/collector/config` | Current collector config |
|
|
| DELETE | `/collector/stream/:id` | Delete a stream |
|
|
|
|
### Stream Auto-Creation
|
|
|
|
Streams are created automatically when you first send logs to them. No pre-configuration needed. Just POST to `/collect/any-name-you-want` and the stream exists.
|
|
|
|
### Stream Naming
|
|
|
|
Stream IDs support: `a-z`, `A-Z`, `0-9`, `-`, `_`, `.`
|
|
|
|
Examples: `my-app`, `web.server.prod`, `auth_service`, `node-1.api`
|