Keviin/LogMaster

Fork 0

Files

Kevin 44cd9ab001

LogMaster CI/CD / build-and-test (push) Has been cancelled

Details

LogMaster CI/CD / docker (push) Has been cancelled

Details

Initial commit: LogMaster with Viewer Collector Monitor modes

2026-05-04 13:50:15 +02:00

21 KiB

Raw Blame History

LogMaster — Complete Documentation

A modular log management platform with four operating modes, a web UI for each, and full API control. Zero external dependencies beyond Node.js.

Architecture
Quick Start
Operating Modes
Viewer
Collector
Monitor
S3 Integration
Configuration
API Reference
GUI Walkthroughs
Example Scenarios
Docker & CI/CD
Keyboard Shortcuts

Architecture

                        ┌──────────────────────────┐
                        │       LOG SOURCES         │
                        ├──────┬──────┬──────┬──────┤
                        │Files │ S3   │ UDP  │ HTTP │
                        │.log  │Bucket│:5140 │POST  │
                        │.gz   │      │      │/collect
                        └──┬───┴──┬───┴──┬───┴──┬───┘
                           │      │      │      │
                           ▼      ▼      ▼      ▼
┌──────────────────────────────────────────────────────────────┐
│                      LOGMASTER SERVER                         │
│                                                              │
│  ┌──────────┐   ┌───────────┐   ┌──────────┐   ┌─────────┐ │
│  │ VIEWER   │   │ COLLECTOR │   │ MONITOR  │   │   ALL   │ │
│  │          │   │           │   │          │   │         │ │
│  │ Read     │   │ Receive   │   │ Scan     │   │ Every   │ │
│  │ Filter   │   │ Store     │   │ Pattern  │   │ feature │ │
│  │ Search   │   │ Rotate    │   │ Alert    │   │ active  │ │
│  │ Export   │   │ Forward   │   │ Notify   │   │         │ │
│  └──────────┘   └───────────┘   └──────────┘   └─────────┘ │
│                                                              │
│  Forward Targets: HTTP │ File │ UDP │ Stream │ Console       │
│  Notifications:   Telegram │ Webhook │ Teams │ Email         │
└──────────────────────────────────────────────────────────────┘

Quick Start

npm install
npm run build                    # Compile TypeScript
node dist-server/server.js       # Start in "both" mode (viewer+collector) on :8080

Open http://localhost:8080.

Start specific modes

# Collector on port 8081
LOG_MODE=collector PORT=8081 node dist-server/server.js

# Viewer on port 9090, reading from collector
LOG_MODE=viewer PORT=9090 REMOTE_COLLECTOR_URL=http://localhost:8081 LOG_SOURCES=./logs/*.log node dist-server/server.js

# Monitor on port 8082
LOG_MODE=monitor PORT=8082 node dist-server/server.js

Operating Modes

Mode	`LOG_MODE=`	What it does	Web UI
Viewer	`viewer`	Read & display logs	Log viewer with filter, search, export
Collector	`collector`	Receive, store, rotate, forward logs	Stream dashboard with live feed
Monitor	`monitor`	Scan files for patterns, send alerts	Rule editor with test & alert history
Both	`both`	Viewer + Collector	Viewer UI with collector active
All	`all`	All three modes	Viewer UI, all features via API

Each mode serves its own dedicated web UI at the root URL.

Viewer

What it does

Reads logs from local files (plain, gzip, bz2, xz, zstd), S3 buckets, and remote collector streams. Displays them in a web UI with filtering, regex search, bookmarks, export, and live tail mode.

GUI walkthrough

Open http://localhost:9090
Sidebar shows all sources grouped by type (Files, S3, Collector Streams, Remote)
Click a source to filter to just that file/stream
Search bar — type to filter. Click .* for regex, Aa for case-sensitive
Level dropdown — filter by ERROR, WARN, INFO, etc.
Tail button — auto-scrolls to new lines. New lines flash blue briefly
Right-click any line for copy, bookmark, filter options
Export — download visible logs as .txt or .json

API examples

# List file sources
curl http://localhost:9090/api/sources

# Add a source
curl -X POST http://localhost:9090/api/sources \
  -H "Content-Type: application/json" \
  -d '{"path": "/var/log/syslog"}'

# Read all logs
curl http://localhost:9090/api/logs

# Read specific source
curl "http://localhost:9090/api/logs?source=/var/log/syslog"

# Read remote collector streams
curl http://localhost:9090/api/remote/streams
curl http://localhost:9090/api/remote/stream/my-app

Configuration

LOG_MODE=viewer
PORT=9090
LOG_SOURCES=/var/log/syslog,/app/logs/*.log
LOG_THEME=dark
LOG_REFRESH_INTERVAL=2000
LOG_TAIL_MODE=true
LOG_MAX_LINES=50000
REMOTE_COLLECTOR_URL=http://collector:8081

Collector

What it does

Receives logs via HTTP POST and UDP, stores them in named streams, rotates files by size or time, compresses rotated files, and forwards to other targets (HTTP, file, UDP, another stream, console).

Sending logs

# Plain text (each line = one log entry)
curl -X POST http://localhost:8081/collect/my-app \
  -d "2026-05-04 ERROR Database connection lost"

# JSON array
curl -X POST http://localhost:8081/collect/my-app/json \
  -H "Content-Type: application/json" \
  -d '["INFO Started", "ERROR Crashed", "WARN Low memory"]'

# UDP (format: streamId|logline)
echo "my-app|2026-05-04 ERROR disk full" | nc -u localhost 5140

Streams are created automatically on first ingest. No pre-configuration needed.

GUI walkthrough

Open http://localhost:8081
Overview shows stream count, total lines, lines/minute
Stream list — click a stream to filter the live feed
Live Feed tab — real-time log lines with color coding (red=ERROR, orange=WARN)
Configuration tab — edit storage, UDP, rotation, compression settings
Forwarding tab — add/remove forward targets

API examples

# List streams
curl http://localhost:8081/collector/streams

# Read a stream
curl http://localhost:8081/collector/stream/my-app

# Read last 10 lines
curl "http://localhost:8081/collector/stream/my-app?tail=10"

# Delete a stream
curl -X DELETE http://localhost:8081/collector/stream/my-app

# Get config
curl http://localhost:8081/collector/config

Log rotation

Rotation triggers on whichever condition is met first:

Trigger	Variable	Default
File size	`COLLECTOR_ROTATION_MAX_SIZE`	5MB (5242880)
File age	`COLLECTOR_ROTATION_MAX_AGE`	10min (600000)

Compression options: none, gzip, bzip2, xz, zstd

Forward targets

# Forward to HTTP + local file + aggregate stream
COLLECTOR_FORWARD_TARGETS=[
  {"type":"http","url":"http://elk:9200/_bulk","format":"json","batchSize":100},
  {"type":"file","path":"/backup/all.log"},
  {"type":"stream","targetStream":"all-logs"},
  {"type":"console"}
]

The stream type copies logs from any stream into another stream on the same collector. Lines are prefixed with [fwd:source-stream].

Configuration

LOG_MODE=collector
PORT=8081
COLLECTOR_UDP_ENABLED=true
COLLECTOR_UDP_PORT=5140
COLLECTOR_STORAGE_DIR=./collector-data
COLLECTOR_MAX_LINES=100000
COLLECTOR_MAX_STREAMS=50
COLLECTOR_PERSIST=true
COLLECTOR_ROTATION_MAX_SIZE=5242880
COLLECTOR_ROTATION_MAX_AGE=600000
COLLECTOR_ROTATION_COMPRESSION=gzip
COLLECTOR_ROTATION_EXTENSION=.log

Monitor

What it does

Scans log files on a schedule, matches regex patterns, and sends notifications via Telegram, Webhook, Microsoft Teams, or Email. Supports time-range filtering, scheduled scans (continuous, daily, hourly), and automatic gzip attachment when matches exceed a threshold.

How scanning works

Every N seconds, the monitor reads all source files for each enabled rule
Only files matching the time range are scanned (today, week, month, last N hours)
Each line is tested against the regex patterns (match any OR match all)
If matches found AND cooldown expired → notifications sent
If matches > maxLinesInMessage → first N lines inline, full results as .gz attachment

GUI walkthrough

Open http://localhost:8082
Click "+ New" to create a rule
Fill in:
- Name, sources (file paths, one per line, globs supported)
- Patterns (one regex per line)
- Time range (today, week, month, hours, all)
- Schedule (continuous, daily at HH:MM, every N hours)
- Max lines in notification, cooldown
Add notification channels — select type, fill fields, click "+ Add Channel"
Click "🧪 Test Notification" to verify the channel works
Click "💾 Save Rule"
Click "🧪 Test Now" to trigger the rule immediately
Alerts appear in the sidebar — click for full detail with all matched lines

Notification channels

Telegram:

Type: telegram
Fields: Bot Token, Chat ID
Get a bot token from @BotFather, chat ID from @userinfobot

Webhook (Slack, Discord, custom):

Type: webhook
Fields: URL

Microsoft Teams:

Type: teams
Fields: Incoming Webhook URL

Email:

Type: email
Fields: SMTP Host, SMTP Port, SMTP User, SMTP Password, To address

API examples

# Create a rule
curl -X POST http://localhost:8082/api/monitor/rules \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Root Access Alert",
    "enabled": true,
    "sources": ["/var/log/auth.log"],
    "patterns": ["root.*login", "su.*root"],
    "patternLogic": "any",
    "timeRange": {"type": "today"},
    "schedule": {"mode": "interval"},
    "notifications": [{"type":"telegram","botToken":"TOKEN","chatId":"CHAT_ID"}],
    "maxLinesInMessage": 20,
    "cooldownMs": 300000
  }'

# List rules
curl http://localhost:8082/api/monitor/rules

# Trigger a rule manually
curl -X POST http://localhost:8082/api/monitor/trigger/rule-id

# View alert history
curl http://localhost:8082/api/monitor/results

# Test a notification channel
curl -X POST http://localhost:8082/api/monitor/test-notification \
  -H "Content-Type: application/json" \
  -d '{"type":"telegram","botToken":"TOKEN","chatId":"CHAT_ID"}'

# Update a rule
curl -X PUT http://localhost:8082/api/monitor/rule/rule-id \
  -H "Content-Type: application/json" \
  -d '{"name": "Updated Name", "enabled": false}'

# Delete a rule
curl -X DELETE http://localhost:8082/api/monitor/rule/rule-id

Configuration

LOG_MODE=monitor
PORT=8082
MONITOR_ENABLED=true
MONITOR_SCAN_INTERVAL=60000
MONITOR_STORAGE_DIR=./monitor-data

Rules can be pre-configured via MONITOR_RULES env var (JSON array) or created via GUI/API at runtime.

S3 Integration

Via GUI

Click ☁️ S3 in the viewer toolbar. Fill in: name, bucket, prefix, access key, secret key, region, optional custom endpoint (for MinIO).

Via environment

S3_ACCESS_KEY_ID=AKIA...
S3_SECRET_ACCESS_KEY=...
S3_REGION=eu-central-1
S3_SOURCE_PROD=my-bucket:logs/production/
S3_SOURCE_STAGING=my-bucket:logs/staging/

Via API

curl -X POST http://localhost:9090/api/s3/add \
  -H "Content-Type: application/json" \
  -d '{"name":"prod","bucket":"my-bucket","prefix":"logs/","accessKeyId":"AKIA...","secretAccessKey":"...","region":"eu-central-1"}'

curl http://localhost:9090/api/s3/sources
curl "http://localhost:9090/api/s3/list?name=prod"
curl "http://localhost:9090/api/s3/read?name=prod&key=logs/app.log"

For S3-compatible services (MinIO, DigitalOcean Spaces), add endpoint:

S3_ENDPOINT=http://minio:9000

Configuration

Priority (highest wins)

Environment variables
logviewer.config.json (editable via GUI)
.env file
Built-in defaults

Full config file example

{
  "mode": "all",
  "port": 8080,
  "viewer": {
    "sources": ["/var/log/syslog", "/app/logs/*.log"],
    "theme": "dark",
    "refreshInterval": 2000,
    "tailMode": true,
    "maxLines": 50000
  },
  "collector": {
    "enabled": true,
    "storageDir": "./collector-data",
    "maxLinesPerStream": 100000,
    "maxStreams": 50,
    "persistToDisk": true,
    "udp": {"enabled": true, "port": 5140},
    "rotation": {"maxSizeBytes": 5242880, "maxAgeMs": 600000, "compression": "gzip", "fileExtension": ".log"},
    "forwardTargets": [{"type": "stream", "targetStream": "all-logs"}]
  },
  "monitor": {
    "enabled": true,
    "scanIntervalMs": 60000,
    "rules": []
  },
  "s3": {
    "sources": {
      "prod": {"bucket": "logs", "prefix": "prod/", "region": "eu-central-1", "accessKeyId": "...", "secretAccessKey": "..."}
    }
  },
  "remoteCollector": {"url": "http://collector:8081"}
}

All environment variables

Variable	Default	Description
`LOG_MODE`	`both`	`viewer`, `collector`, `monitor`, `both`, `all`
`PORT`	`8080`	HTTP port
`HOST`	`0.0.0.0`	Bind address
`LOG_SOURCES`	—	Comma-separated file paths/globs
`LOG_THEME`	`dark`	`dark` or `light`
`LOG_REFRESH_INTERVAL`	`2000`	Auto-refresh ms
`LOG_TAIL_MODE`	`true`	Auto-scroll
`LOG_MAX_LINES`	`50000`	Max lines per file
`COLLECTOR_UDP_ENABLED`	`true`	UDP receiver
`COLLECTOR_UDP_PORT`	`5140`	UDP port
`COLLECTOR_STORAGE_DIR`	`./collector-data`	Storage path
`COLLECTOR_PERSIST`	`true`	Write to disk
`COLLECTOR_ROTATION_MAX_SIZE`	`5242880`	Rotate at bytes
`COLLECTOR_ROTATION_MAX_AGE`	`600000`	Rotate at ms
`COLLECTOR_ROTATION_COMPRESSION`	`gzip`	`none`/`gzip`/`bzip2`/`xz`/`zstd`
`COLLECTOR_FORWARD_TARGETS`	`[]`	JSON array
`MONITOR_ENABLED`	`true`	Enable scanning
`MONITOR_SCAN_INTERVAL`	`60000`	Scan interval ms
`MONITOR_RULES`	`[]`	JSON array of rules
`REMOTE_COLLECTOR_URL`	—	Remote collector URL
`S3_ACCESS_KEY_ID`	—	S3 access key
`S3_SECRET_ACCESS_KEY`	—	S3 secret key
`S3_REGION`	`us-east-1`	S3 region
`S3_ENDPOINT`	—	Custom S3 endpoint

API Reference

General

Method	Path	Description
GET	`/api/mode`	Current mode info
GET	`/api/config`	Active configuration
POST	`/api/config/save`	Save config to file

Viewer

Method	Path	Description
GET	`/api/sources`	List file sources
POST	`/api/sources`	Add source `{"path":"..."}`
DELETE	`/api/sources?path=...`	Remove source
GET	`/api/logs?source=...`	Read logs
GET	`/api/remote/streams`	Remote collector streams
GET	`/api/remote/stream/:id`	Read remote stream

Collector

Method	Path	Description
POST	`/collect/:id`	Ingest plain text
POST	`/collect/:id/json`	Ingest JSON
GET	`/collector/streams`	List streams
GET	`/collector/stream/:id?tail=N`	Read stream
GET	`/collector/config`	Collector config
DELETE	`/collector/stream/:id`	Delete stream

Monitor

Method	Path	Description
GET	`/api/monitor/rules`	List rules
POST	`/api/monitor/rules`	Create rule
PUT	`/api/monitor/rule/:id`	Update rule
DELETE	`/api/monitor/rule/:id`	Delete rule
POST	`/api/monitor/trigger/:id`	Trigger rule now
GET	`/api/monitor/results`	Alert history
POST	`/api/monitor/test-notification`	Test a channel

S3

Method	Path	Description
POST	`/api/s3/add`	Add S3 source
GET	`/api/s3/sources`	List S3 sources
GET	`/api/s3/list?name=...`	List S3 objects
GET	`/api/s3/read?name=...&key=...`	Read S3 object

GUI Walkthroughs

Viewer: Search for errors in the last hour

Open http://localhost:9090
Select "Error" from the level dropdown
Type a search term or regex in the search bar
Click .* to enable regex mode
Results update live. Click 💾 Export to download

Collector: Watch live logs from multiple apps

Open http://localhost:8081
The Live Feed tab shows all incoming logs in real-time
Click a stream in the sidebar to filter
Click ⏸ Pause to freeze the feed
Go to ⚙️ Configuration to change rotation/compression settings

Monitor: Set up a daily security scan

Open http://localhost:8082
Click "+ New"
Name: "Daily Security Audit"
Sources: /var/log/auth.log (one per line)
Patterns: root.*login, sudo.*COMMAND, Failed password (one per line)
Time Range: Today
Schedule: Daily at 07:00
Add Telegram notification: paste bot token and chat ID
Click "🧪 Test Notification" to verify
Click "💾 Save Rule"
Click "🧪 Test Now" to see immediate results

Example Scenarios

1. Development: All-in-one

LOG_MODE=both PORT=8080 LOG_SOURCES=./logs/*.log node dist-server/server.js

View logs + receive from apps on one port.

2. Production: Separate services

# Server A: Collector
LOG_MODE=collector PORT=8081 COLLECTOR_UDP_PORT=5140 \
  COLLECTOR_ROTATION_COMPRESSION=gzip \
  COLLECTOR_FORWARD_TARGETS='[{"type":"stream","targetStream":"all-logs"}]' \
  node dist-server/server.js

# Server B: Viewer
LOG_MODE=viewer PORT=9090 REMOTE_COLLECTOR_URL=http://serverA:8081 \
  node dist-server/server.js

# Server C: Monitor
LOG_MODE=monitor PORT=8082 MONITOR_SCAN_INTERVAL=30000 \
  node dist-server/server.js

3. Security monitoring with Telegram alerts

LOG_MODE=monitor PORT=8082 MONITOR_SCAN_INTERVAL=10000 \
  MONITOR_RULES='[{"id":"sec","name":"Security","enabled":true,"sources":["/var/log/auth.log"],"patterns":["root","sudo","Failed"],"patternLogic":"any","timeRange":{"type":"today"},"notifications":[{"type":"telegram","botToken":"YOUR_TOKEN","chatId":"YOUR_CHAT"}],"maxLinesInMessage":20,"cooldownMs":60000}]' \
  node dist-server/server.js

4. Log aggregation with S3 archive

LOG_MODE=both PORT=8080 \
  S3_ACCESS_KEY_ID=AKIA... S3_SECRET_ACCESS_KEY=... S3_REGION=eu-central-1 \
  S3_SOURCE_ARCHIVE=my-bucket:logs/ \
  COLLECTOR_FORWARD_TARGETS='[{"type":"file","path":"/archive/all.log"}]' \
  node dist-server/server.js

Docker & CI/CD

Build and run

docker build -t logmaster .

# Collector
docker run -d -p 8081:8080 -p 5140:5140/udp -e LOG_MODE=collector logmaster

# Viewer
docker run -d -p 9090:8080 -e LOG_MODE=viewer -e REMOTE_COLLECTOR_URL=http://collector:8080 logmaster

# Monitor
docker run -d -p 8082:8080 -e LOG_MODE=monitor -v /var/log:/logs:ro logmaster

Docker Compose

services:
  collector:
    build: .
    ports: ["8081:8080", "5140:5140/udp"]
    environment:
      LOG_MODE: collector
      COLLECTOR_UDP_PORT: "5140"
    volumes:
      - collector-data:/app/collector-data

  viewer:
    build: .
    ports: ["9090:8080"]
    environment:
      LOG_MODE: viewer
      REMOTE_COLLECTOR_URL: http://collector:8080
    depends_on: [collector]

  monitor:
    build: .
    ports: ["8082:8080"]
    environment:
      LOG_MODE: monitor
      MONITOR_SCAN_INTERVAL: "30000"
    volumes:
      - /var/log:/logs:ro

volumes:
  collector-data:

CI/CD (Gitea Actions)

The .gitea/workflows/ci.yaml pipeline:

Installs dependencies
Compiles TypeScript
Generates sample logs
Starts all 3 services
Runs the full integration test suite (41 tests)
Builds Docker image
Pushes to Gitea container registry

To enable: add a Gitea Actions runner and set REGISTRY_USER / REGISTRY_PASSWORD secrets.

Keyboard Shortcuts

Key	Action
`Ctrl+F`	Search
`Ctrl+Shift+F`	Regex search
`Ctrl+T`	Toggle tail mode
`Ctrl+W`	Toggle word wrap
`Ctrl+G`	Go to line
`Ctrl+Shift+T`	Toggle theme
`Ctrl+=` / `Ctrl+-`	Zoom in/out
`Ctrl+0`	Reset zoom
`Home` / `End`	Scroll top/bottom
`Escape`	Clear search / close modals

Supported Log Formats

Format	Extensions	Decompression
Plain text	`.log`, `.txt`, no extension	Direct read
Gzip	`.gz`	Node.js zlib (built-in)
Bzip2	`.bz2`	`bzcat` CLI
XZ	`.xz`	`xzcat` CLI
Zstandard	`.zst`	`zstdcat` CLI

License

MIT

21 KiB Raw Blame History

LogMaster — Complete Documentation

Table of Contents

Architecture

Quick Start

Start specific modes

Operating Modes

Viewer

What it does

GUI walkthrough

API examples

Configuration

Collector

What it does

Sending logs

GUI walkthrough

API examples

Log rotation

Forward targets

Configuration

Monitor

What it does

How scanning works

GUI walkthrough

Notification channels

API examples

Configuration

S3 Integration

Via GUI

Via environment

Via API

Configuration

Priority (highest wins)

Full config file example

All environment variables

API Reference

General

Viewer

Collector

Monitor

S3

GUI Walkthroughs

Viewer: Search for errors in the last hour

Collector: Watch live logs from multiple apps

Monitor: Set up a daily security scan

Example Scenarios

1. Development: All-in-one

2. Production: Separate services

3. Security monitoring with Telegram alerts

4. Log aggregation with S3 archive

Docker & CI/CD

Build and run

Docker Compose

CI/CD (Gitea Actions)

Keyboard Shortcuts

Supported Log Formats

License

21 KiB

Raw Blame History