Keviin/LogMaster

Fork 0

Files

Kevin 44cd9ab001

LogMaster CI/CD / build-and-test (push) Has been cancelled

Details

LogMaster CI/CD / docker (push) Has been cancelled

Details

Initial commit: LogMaster with Viewer Collector Monitor modes

2026-05-04 13:50:15 +02:00

12 KiB

Raw Blame History

📡 Log Collector — User Guide

Overview

The Log Collector receives logs from applications via HTTP and UDP, stores them in named streams, rotates log files based on size/time, and optionally forwards them to other systems (databases, other collectors, files).

Starting the Collector

Option 1: Environment File

cp .env.collector .env
node dist-server/server.js

Option 2: Inline

LOG_MODE=collector PORT=8081 COLLECTOR_UDP_PORT=5140 node dist-server/server.js

Option 3: Docker

docker run -d -p 8081:8080 -p 5140:5140/udp \
  -e LOG_MODE=collector \
  -e COLLECTOR_UDP_PORT=5140 \
  -e COLLECTOR_ROTATION_COMPRESSION=gzip \
  -v ./collector-data:/app/collector-data \
  logviewer

Option 4: Config File

{
  "mode": "collector",
  "port": 8081,
  "collector": {
    "enabled": true,
    "storageDir": "./collector-data",
    "maxLinesPerStream": 100000,
    "maxStreams": 50,
    "persistToDisk": true,
    "udp": { "enabled": true, "port": 5140 },
    "rotation": {
      "maxSizeBytes": 5242880,
      "maxAgeMs": 600000,
      "compression": "gzip",
      "fileExtension": ".log"
    },
    "forwardTargets": []
  }
}

Sending Logs to the Collector

HTTP POST (Plain Text)

Each line in the body becomes one log entry:

curl -X POST http://localhost:8081/collect/my-app \
  -d "2026-04-30T10:00:01Z INFO Application started"

Multiple lines:

curl -X POST http://localhost:8081/collect/my-app -d "line 1
line 2
line 3"

HTTP POST (JSON)

Send an array of strings or objects:

curl -X POST http://localhost:8081/collect/my-app/json \
  -H "Content-Type: application/json" \
  -d '["INFO App started", "ERROR Crash!", {"ts":"2026-04-30","level":"WARN","msg":"low memory"}]'

UDP

Format: streamId|logline — the part before | is the stream name:

echo "my-app|2026-04-30 ERROR database timeout" | nc -u localhost 5140

Without stream ID (auto-named by sender IP):

echo "2026-04-30 INFO hello from UDP" | nc -u localhost 5140

From Applications

Node.js:

const http = require('http');
function log(stream, msg) {
  const req = http.request({ hostname:'localhost', port:8081, path:`/collect/${stream}`, method:'POST' });
  req.write(`${new Date().toISOString()} ${msg}`);
  req.end();
}
log('my-app', 'INFO Server started on port 3000');

Python:

import requests, datetime
def log(stream, msg):
    ts = datetime.datetime.now().isoformat()
    requests.post(f'http://localhost:8081/collect/{stream}', data=f'{ts} {msg}')

log('my-app', 'INFO Processing complete')

Bash (continuous pipe):

tail -f /var/log/myapp.log | while read line; do
  curl -s -X POST http://localhost:8081/collect/myapp -d "$line"
done

Test Script (included)

Sends random logs every 500ms:

node send-test-logs.js <stream-name> <collector-url>
node send-test-logs.js demo-app http://localhost:8081

GUI Guide

Layout

┌─────────────────────────────────────────────────────────────┐
│  📡 Log Collector [ACTIVE]                    Uptime: 5m 32s│
├──────────────┬──────────────────────────────────────────────┤
│              │  [🔴 Live Feed] [⚙️ Configuration] [🔀 Fwd]  │
│  📊 Overview │                                              │
│  ┌─────────┐ │  Live Feed:                                  │
│  │ 3       │ │  [demo-app] 10:00:01 INFO App started        │
│  │ Streams │ │  [demo-app] 10:00:02 ERROR DB timeout  ← red │
│  ├─────────┤ │  [web-srv] 10:00:03 WARN Slow request ← orange
│  │ 1.2k    │ │  [demo-app] 10:00:04 INFO Request OK         │
│  │ Lines   │ │                                              │
│  ├─────────┤ │  New lines appear with animation             │
│  │ 120     │ │                                              │
│  │ /min    │ │                                              │
│  └─────────┘ │                                              │
│              │                                              │
│  📡 Streams  │                                              │
│  ┌──────────┐│                                              │
│  │ demo-app ││                                              │
│  │ 450 lines││                                              │
│  │ 2s ago   ││                                              │
│  ├──────────┤│                                              │
│  │ web-srv  ││                                              │
│  │ 120 lines││                                              │
│  └──────────┘│                                              │
│              │                                              │
│  🔗 Endpoints                                               │
│  POST /collect/<id>                                         │
│  UDP :5140   │                                              │
└──────────────┴──────────────────────────────────────────────┘

Tabs

🔴 Live Feed

Shows incoming log lines in real-time
Color-coded by level (red=ERROR, orange=WARN, blue=INFO)
New lines animate in from the left
Click a stream in the sidebar to filter the feed
Pause/Resume button to freeze the feed
Clear button to reset

⚙️ Configuration

Edit all collector settings via the GUI
Changes are saved to logviewer.config.json
Settings: storage dir, max streams, max lines, UDP port, rotation size/age/compression

🔀 Forwarding

View active forward targets
Add new targets (HTTP, File, UDP, Console)
Configure batch size and flush interval

Configuration Reference

Environment Variables

Variable	Default	Description
`LOG_MODE`	`both`	Set to `collector`
`PORT`	`8080`	HTTP port
`COLLECTOR_UDP_ENABLED`	`true`	Enable/disable UDP receiver
`COLLECTOR_UDP_PORT`	`5140`	UDP listen port
`COLLECTOR_STORAGE_DIR`	`./collector-data`	Where to store log files
`COLLECTOR_MAX_LINES`	`100000`	Max lines kept in memory per stream
`COLLECTOR_MAX_STREAMS`	`50`	Max number of streams
`COLLECTOR_PERSIST`	`true`	Write logs to disk
`COLLECTOR_ROTATION_MAX_SIZE`	`5242880`	Rotate after N bytes (default 5MB)
`COLLECTOR_ROTATION_MAX_AGE`	`600000`	Rotate after N ms (default 10min)
`COLLECTOR_ROTATION_COMPRESSION`	`gzip`	`none`, `gzip`, `bzip2`, `xz`, `zstd`
`COLLECTOR_ROTATION_EXTENSION`	`.log`	File extension for active log
`COLLECTOR_FORWARD_TARGETS`	`[]`	JSON array of forward targets

Log Rotation

Rotation triggers on whichever condition is met first:

Condition	Variable	Example
File size exceeds limit	`COLLECTOR_ROTATION_MAX_SIZE=10485760`	Rotate at 10MB
File age exceeds limit	`COLLECTOR_ROTATION_MAX_AGE=3600000`	Rotate every hour

Rotated files are named with timestamp and compressed:

collector-data/
├── my-app.log                              ← current (active)
├── my-app_2026-04-30T10-00-00-000Z.log.gz  ← rotated + gzipped
├── my-app_2026-04-29T22-00-00-000Z.log.gz
└── _streams.json                           ← metadata

Forward Targets

Forward targets receive copies of all ingested logs. Configure via environment or GUI.

HTTP Target — POST logs to another service:

{"type":"http","url":"http://elasticsearch:9200/_bulk","format":"json","batchSize":200,"flushIntervalMs":5000}

File Target — Append to a file:

{"type":"file","path":"/backup/all-logs.txt"}

UDP Target — Forward via UDP (e.g., to syslog):

{"type":"udp","host":"syslog.internal","port":514}

Stream Target — Forward to another stream on the same collector:

{"type":"stream","targetStream":"all-logs"}

This copies all ingested lines from any stream into the all-logs stream. Lines are prefixed with [fwd:source-stream] to identify the origin. Useful for creating an aggregated "all logs" stream.

Console Target — Print to stdout (for debugging):

{"type":"console"}

Multiple targets:

COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://elk:9200/_bulk","format":"json","batchSize":100},{"type":"file","path":"/backup/all.log"},{"type":"console"}]

Example Configurations

Minimal Collector

LOG_MODE=collector
PORT=8081

Production Collector (large scale)

LOG_MODE=collector
PORT=8081
COLLECTOR_UDP_PORT=5140
COLLECTOR_STORAGE_DIR=/data/logs
COLLECTOR_MAX_LINES=500000
COLLECTOR_MAX_STREAMS=200
COLLECTOR_ROTATION_MAX_SIZE=52428800
COLLECTOR_ROTATION_MAX_AGE=3600000
COLLECTOR_ROTATION_COMPRESSION=zstd
COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://elasticsearch:9200/logs/_bulk","format":"json","batchSize":500,"flushIntervalMs":3000}]

Collector with multiple forward targets

LOG_MODE=collector
PORT=8081
COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://loki:3100/loki/api/v1/push","format":"json","batchSize":100},{"type":"file","path":"/archive/all-logs.txt"},{"type":"udp","host":"siem.internal","port":514}]

Collector with stream aggregation (self-forward)

LOG_MODE=collector
PORT=8081
COLLECTOR_FORWARD_TARGETS=[{"type":"stream","targetStream":"all-logs"},{"type":"console"}]

Every log sent to any stream (e.g., web-app, auth-service) is also copied into the all-logs stream. Lines are prefixed with [fwd:web-app] etc. so you can still tell where they came from.

Docker Compose: Collector cluster

services:
  collector-1:
    image: logviewer
    ports: ["8081:8080", "5140:5140/udp"]
    environment:
      LOG_MODE: collector
      COLLECTOR_UDP_PORT: "5140"
      COLLECTOR_ROTATION_COMPRESSION: gzip
      COLLECTOR_FORWARD_TARGETS: '[{"type":"http","url":"http://central:8080/collect/region-eu","format":"json"}]'
    volumes:
      - ./data-eu:/app/collector-data

  collector-2:
    image: logviewer
    ports: ["8082:8080", "5141:5140/udp"]
    environment:
      LOG_MODE: collector
      COLLECTOR_UDP_PORT: "5140"
      COLLECTOR_FORWARD_TARGETS: '[{"type":"http","url":"http://central:8080/collect/region-us","format":"json"}]'
    volumes:
      - ./data-us:/app/collector-data

  central:
    image: logviewer
    ports: ["9090:8080"]
    environment:
      LOG_MODE: both

API Reference

Method	Path	Description
POST	`/collect/:streamId`	Ingest plain text (one line per line)
POST	`/collect/:streamId/json`	Ingest JSON array
GET	`/collector/streams`	List all streams
GET	`/collector/stream/:id?tail=N`	Read stream (optional: last N lines)
GET	`/collector/config`	Current collector config
DELETE	`/collector/stream/:id`	Delete a stream

Stream Auto-Creation

Streams are created automatically when you first send logs to them. No pre-configuration needed. Just POST to /collect/any-name-you-want and the stream exists.

Stream Naming

Stream IDs support: a-z, A-Z, 0-9, -, _, .

Examples: my-app, web.server.prod, auth_service, node-1.api

12 KiB Raw Blame History