# LogMaster — Complete Documentation

A modular log management platform with four operating modes, a web UI for each, and full API control. Zero external dependencies beyond Node.js.

---

## Table of Contents

- [Architecture](#architecture)
- [Quick Start](#quick-start)
- [Operating Modes](#operating-modes)
- [Viewer](#viewer)
- [Collector](#collector)
- [Monitor](#monitor)
- [S3 Integration](#s3-integration)
- [Configuration](#configuration)
- [API Reference](#api-reference)
- [GUI Walkthroughs](#gui-walkthroughs)
- [Example Scenarios](#example-scenarios)
- [Docker & CI/CD](#docker--cicd)
- [Keyboard Shortcuts](#keyboard-shortcuts)

---

## Architecture

```
                        ┌──────────────────────────┐
                        │       LOG SOURCES         │
                        ├──────┬──────┬──────┬──────┤
                        │Files │ S3   │ UDP  │ HTTP │
                        │.log  │Bucket│:5140 │POST  │
                        │.gz   │      │      │/collect
                        └──┬───┴──┬───┴──┬───┴──┬───┘
                           │      │      │      │
                           ▼      ▼      ▼      ▼
┌──────────────────────────────────────────────────────────────┐
│                      LOGMASTER SERVER                         │
│                                                              │
│  ┌──────────┐   ┌───────────┐   ┌──────────┐   ┌─────────┐ │
│  │ VIEWER   │   │ COLLECTOR │   │ MONITOR  │   │   ALL   │ │
│  │          │   │           │   │          │   │         │ │
│  │ Read     │   │ Receive   │   │ Scan     │   │ Every   │ │
│  │ Filter   │   │ Store     │   │ Pattern  │   │ feature │ │
│  │ Search   │   │ Rotate    │   │ Alert    │   │ active  │ │
│  │ Export   │   │ Forward   │   │ Notify   │   │         │ │
│  └──────────┘   └───────────┘   └──────────┘   └─────────┘ │
│                                                              │
│  Forward Targets: HTTP │ File │ UDP │ Stream │ Console       │
│  Notifications:   Telegram │ Webhook │ Teams │ Email         │
└──────────────────────────────────────────────────────────────┘
```

---

## Quick Start

```bash
npm install
npm run build                    # Compile TypeScript
node dist-server/server.js       # Start in "both" mode (viewer+collector) on :8080
```

Open http://localhost:8080.

### Start specific modes

```bash
# Collector on port 8081
LOG_MODE=collector PORT=8081 node dist-server/server.js

# Viewer on port 9090, reading from collector
LOG_MODE=viewer PORT=9090 REMOTE_COLLECTOR_URL=http://localhost:8081 LOG_SOURCES=./logs/*.log node dist-server/server.js

# Monitor on port 8082
LOG_MODE=monitor PORT=8082 node dist-server/server.js
```

---

## Operating Modes

| Mode | `LOG_MODE=` | What it does | Web UI |
|------|-------------|-------------|--------|
| Viewer | `viewer` | Read & display logs | Log viewer with filter, search, export |
| Collector | `collector` | Receive, store, rotate, forward logs | Stream dashboard with live feed |
| Monitor | `monitor` | Scan files for patterns, send alerts | Rule editor with test & alert history |
| Both | `both` | Viewer + Collector | Viewer UI with collector active |
| All | `all` | All three modes | Viewer UI, all features via API |

Each mode serves its own dedicated web UI at the root URL.

---

## Viewer

### What it does

Reads logs from local files (plain, gzip, bz2, xz, zstd), S3 buckets, and remote collector streams. Displays them in a web UI with filtering, regex search, bookmarks, export, and live tail mode.

### GUI walkthrough

1. **Open** http://localhost:9090
2. **Sidebar** shows all sources grouped by type (Files, S3, Collector Streams, Remote)
3. **Click a source** to filter to just that file/stream
4. **Search bar** — type to filter. Click `.*` for regex, `Aa` for case-sensitive
5. **Level dropdown** — filter by ERROR, WARN, INFO, etc.
6. **Tail button** — auto-scrolls to new lines. New lines flash blue briefly
7. **Right-click** any line for copy, bookmark, filter options
8. **Export** — download visible logs as .txt or .json

### API examples

```bash
# List file sources
curl http://localhost:9090/api/sources

# Add a source
curl -X POST http://localhost:9090/api/sources \
  -H "Content-Type: application/json" \
  -d '{"path": "/var/log/syslog"}'

# Read all logs
curl http://localhost:9090/api/logs

# Read specific source
curl "http://localhost:9090/api/logs?source=/var/log/syslog"

# Read remote collector streams
curl http://localhost:9090/api/remote/streams
curl http://localhost:9090/api/remote/stream/my-app
```

### Configuration

```env
LOG_MODE=viewer
PORT=9090
LOG_SOURCES=/var/log/syslog,/app/logs/*.log
LOG_THEME=dark
LOG_REFRESH_INTERVAL=2000
LOG_TAIL_MODE=true
LOG_MAX_LINES=50000
REMOTE_COLLECTOR_URL=http://collector:8081
```

---

## Collector

### What it does

Receives logs via HTTP POST and UDP, stores them in named streams, rotates files by size or time, compresses rotated files, and forwards to other targets (HTTP, file, UDP, another stream, console).

### Sending logs

```bash
# Plain text (each line = one log entry)
curl -X POST http://localhost:8081/collect/my-app \
  -d "2026-05-04 ERROR Database connection lost"

# JSON array
curl -X POST http://localhost:8081/collect/my-app/json \
  -H "Content-Type: application/json" \
  -d '["INFO Started", "ERROR Crashed", "WARN Low memory"]'

# UDP (format: streamId|logline)
echo "my-app|2026-05-04 ERROR disk full" | nc -u localhost 5140
```

Streams are created automatically on first ingest. No pre-configuration needed.

### GUI walkthrough

1. **Open** http://localhost:8081
2. **Overview** shows stream count, total lines, lines/minute
3. **Stream list** — click a stream to filter the live feed
4. **Live Feed tab** — real-time log lines with color coding (red=ERROR, orange=WARN)
5. **Configuration tab** — edit storage, UDP, rotation, compression settings
6. **Forwarding tab** — add/remove forward targets

### API examples

```bash
# List streams
curl http://localhost:8081/collector/streams

# Read a stream
curl http://localhost:8081/collector/stream/my-app

# Read last 10 lines
curl "http://localhost:8081/collector/stream/my-app?tail=10"

# Delete a stream
curl -X DELETE http://localhost:8081/collector/stream/my-app

# Get config
curl http://localhost:8081/collector/config
```

### Log rotation

Rotation triggers on whichever condition is met first:

| Trigger | Variable | Default |
|---------|----------|---------|
| File size | `COLLECTOR_ROTATION_MAX_SIZE` | 5MB (5242880) |
| File age | `COLLECTOR_ROTATION_MAX_AGE` | 10min (600000) |

Compression options: `none`, `gzip`, `bzip2`, `xz`, `zstd`

### Forward targets

```env
# Forward to HTTP + local file + aggregate stream
COLLECTOR_FORWARD_TARGETS=[
  {"type":"http","url":"http://elk:9200/_bulk","format":"json","batchSize":100},
  {"type":"file","path":"/backup/all.log"},
  {"type":"stream","targetStream":"all-logs"},
  {"type":"console"}
]
```

The `stream` type copies logs from any stream into another stream on the same collector. Lines are prefixed with `[fwd:source-stream]`.

### Configuration

```env
LOG_MODE=collector
PORT=8081
COLLECTOR_UDP_ENABLED=true
COLLECTOR_UDP_PORT=5140
COLLECTOR_STORAGE_DIR=./collector-data
COLLECTOR_MAX_LINES=100000
COLLECTOR_MAX_STREAMS=50
COLLECTOR_PERSIST=true
COLLECTOR_ROTATION_MAX_SIZE=5242880
COLLECTOR_ROTATION_MAX_AGE=600000
COLLECTOR_ROTATION_COMPRESSION=gzip
COLLECTOR_ROTATION_EXTENSION=.log
```

---

## Monitor

### What it does

Scans log files on a schedule, matches regex patterns, and sends notifications via Telegram, Webhook, Microsoft Teams, or Email. Supports time-range filtering, scheduled scans (continuous, daily, hourly), and automatic gzip attachment when matches exceed a threshold.

### How scanning works

1. Every N seconds, the monitor reads all source files for each enabled rule
2. Only files matching the time range are scanned (today, week, month, last N hours)
3. Each line is tested against the regex patterns (match any OR match all)
4. If matches found AND cooldown expired → notifications sent
5. If matches > `maxLinesInMessage` → first N lines inline, full results as .gz attachment

### GUI walkthrough

1. **Open** http://localhost:8082
2. **Click "+ New"** to create a rule
3. **Fill in:**
   - Name, sources (file paths, one per line, globs supported)
   - Patterns (one regex per line)
   - Time range (today, week, month, hours, all)
   - Schedule (continuous, daily at HH:MM, every N hours)
   - Max lines in notification, cooldown
4. **Add notification channels** — select type, fill fields, click "+ Add Channel"
5. **Click "🧪 Test Notification"** to verify the channel works
6. **Click "💾 Save Rule"**
7. **Click "🧪 Test Now"** to trigger the rule immediately
8. **Alerts** appear in the sidebar — click for full detail with all matched lines

### Notification channels

**Telegram:**
- Type: `telegram`
- Fields: Bot Token, Chat ID
- Get a bot token from @BotFather, chat ID from @userinfobot

**Webhook (Slack, Discord, custom):**
- Type: `webhook`
- Fields: URL

**Microsoft Teams:**
- Type: `teams`
- Fields: Incoming Webhook URL

**Email:**
- Type: `email`
- Fields: SMTP Host, SMTP Port, SMTP User, SMTP Password, To address

### API examples

```bash
# Create a rule
curl -X POST http://localhost:8082/api/monitor/rules \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Root Access Alert",
    "enabled": true,
    "sources": ["/var/log/auth.log"],
    "patterns": ["root.*login", "su.*root"],
    "patternLogic": "any",
    "timeRange": {"type": "today"},
    "schedule": {"mode": "interval"},
    "notifications": [{"type":"telegram","botToken":"TOKEN","chatId":"CHAT_ID"}],
    "maxLinesInMessage": 20,
    "cooldownMs": 300000
  }'

# List rules
curl http://localhost:8082/api/monitor/rules

# Trigger a rule manually
curl -X POST http://localhost:8082/api/monitor/trigger/rule-id

# View alert history
curl http://localhost:8082/api/monitor/results

# Test a notification channel
curl -X POST http://localhost:8082/api/monitor/test-notification \
  -H "Content-Type: application/json" \
  -d '{"type":"telegram","botToken":"TOKEN","chatId":"CHAT_ID"}'

# Update a rule
curl -X PUT http://localhost:8082/api/monitor/rule/rule-id \
  -H "Content-Type: application/json" \
  -d '{"name": "Updated Name", "enabled": false}'

# Delete a rule
curl -X DELETE http://localhost:8082/api/monitor/rule/rule-id
```

### Configuration

```env
LOG_MODE=monitor
PORT=8082
MONITOR_ENABLED=true
MONITOR_SCAN_INTERVAL=60000
MONITOR_STORAGE_DIR=./monitor-data
```

Rules can be pre-configured via `MONITOR_RULES` env var (JSON array) or created via GUI/API at runtime.

---

## S3 Integration

### Via GUI

Click **☁️ S3** in the viewer toolbar. Fill in: name, bucket, prefix, access key, secret key, region, optional custom endpoint (for MinIO).

### Via environment

```env
S3_ACCESS_KEY_ID=AKIA...
S3_SECRET_ACCESS_KEY=...
S3_REGION=eu-central-1
S3_SOURCE_PROD=my-bucket:logs/production/
S3_SOURCE_STAGING=my-bucket:logs/staging/
```

### Via API

```bash
curl -X POST http://localhost:9090/api/s3/add \
  -H "Content-Type: application/json" \
  -d '{"name":"prod","bucket":"my-bucket","prefix":"logs/","accessKeyId":"AKIA...","secretAccessKey":"...","region":"eu-central-1"}'

curl http://localhost:9090/api/s3/sources
curl "http://localhost:9090/api/s3/list?name=prod"
curl "http://localhost:9090/api/s3/read?name=prod&key=logs/app.log"
```

For S3-compatible services (MinIO, DigitalOcean Spaces), add `endpoint`:
```env
S3_ENDPOINT=http://minio:9000
```

---

## Configuration

### Priority (highest wins)

1. Environment variables
2. `logviewer.config.json` (editable via GUI)
3. `.env` file
4. Built-in defaults

### Full config file example

```json
{
  "mode": "all",
  "port": 8080,
  "viewer": {
    "sources": ["/var/log/syslog", "/app/logs/*.log"],
    "theme": "dark",
    "refreshInterval": 2000,
    "tailMode": true,
    "maxLines": 50000
  },
  "collector": {
    "enabled": true,
    "storageDir": "./collector-data",
    "maxLinesPerStream": 100000,
    "maxStreams": 50,
    "persistToDisk": true,
    "udp": {"enabled": true, "port": 5140},
    "rotation": {"maxSizeBytes": 5242880, "maxAgeMs": 600000, "compression": "gzip", "fileExtension": ".log"},
    "forwardTargets": [{"type": "stream", "targetStream": "all-logs"}]
  },
  "monitor": {
    "enabled": true,
    "scanIntervalMs": 60000,
    "rules": []
  },
  "s3": {
    "sources": {
      "prod": {"bucket": "logs", "prefix": "prod/", "region": "eu-central-1", "accessKeyId": "...", "secretAccessKey": "..."}
    }
  },
  "remoteCollector": {"url": "http://collector:8081"}
}
```

### All environment variables

| Variable | Default | Description |
|----------|---------|-------------|
| `LOG_MODE` | `both` | `viewer`, `collector`, `monitor`, `both`, `all` |
| `PORT` | `8080` | HTTP port |
| `HOST` | `0.0.0.0` | Bind address |
| `LOG_SOURCES` | — | Comma-separated file paths/globs |
| `LOG_THEME` | `dark` | `dark` or `light` |
| `LOG_REFRESH_INTERVAL` | `2000` | Auto-refresh ms |
| `LOG_TAIL_MODE` | `true` | Auto-scroll |
| `LOG_MAX_LINES` | `50000` | Max lines per file |
| `COLLECTOR_UDP_ENABLED` | `true` | UDP receiver |
| `COLLECTOR_UDP_PORT` | `5140` | UDP port |
| `COLLECTOR_STORAGE_DIR` | `./collector-data` | Storage path |
| `COLLECTOR_PERSIST` | `true` | Write to disk |
| `COLLECTOR_ROTATION_MAX_SIZE` | `5242880` | Rotate at bytes |
| `COLLECTOR_ROTATION_MAX_AGE` | `600000` | Rotate at ms |
| `COLLECTOR_ROTATION_COMPRESSION` | `gzip` | `none`/`gzip`/`bzip2`/`xz`/`zstd` |
| `COLLECTOR_FORWARD_TARGETS` | `[]` | JSON array |
| `MONITOR_ENABLED` | `true` | Enable scanning |
| `MONITOR_SCAN_INTERVAL` | `60000` | Scan interval ms |
| `MONITOR_RULES` | `[]` | JSON array of rules |
| `REMOTE_COLLECTOR_URL` | — | Remote collector URL |
| `S3_ACCESS_KEY_ID` | — | S3 access key |
| `S3_SECRET_ACCESS_KEY` | — | S3 secret key |
| `S3_REGION` | `us-east-1` | S3 region |
| `S3_ENDPOINT` | — | Custom S3 endpoint |

---

## API Reference

### General
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/mode` | Current mode info |
| GET | `/api/config` | Active configuration |
| POST | `/api/config/save` | Save config to file |

### Viewer
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/sources` | List file sources |
| POST | `/api/sources` | Add source `{"path":"..."}` |
| DELETE | `/api/sources?path=...` | Remove source |
| GET | `/api/logs?source=...` | Read logs |
| GET | `/api/remote/streams` | Remote collector streams |
| GET | `/api/remote/stream/:id` | Read remote stream |

### Collector
| Method | Path | Description |
|--------|------|-------------|
| POST | `/collect/:id` | Ingest plain text |
| POST | `/collect/:id/json` | Ingest JSON |
| GET | `/collector/streams` | List streams |
| GET | `/collector/stream/:id?tail=N` | Read stream |
| GET | `/collector/config` | Collector config |
| DELETE | `/collector/stream/:id` | Delete stream |

### Monitor
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/monitor/rules` | List rules |
| POST | `/api/monitor/rules` | Create rule |
| PUT | `/api/monitor/rule/:id` | Update rule |
| DELETE | `/api/monitor/rule/:id` | Delete rule |
| POST | `/api/monitor/trigger/:id` | Trigger rule now |
| GET | `/api/monitor/results` | Alert history |
| POST | `/api/monitor/test-notification` | Test a channel |

### S3
| Method | Path | Description |
|--------|------|-------------|
| POST | `/api/s3/add` | Add S3 source |
| GET | `/api/s3/sources` | List S3 sources |
| GET | `/api/s3/list?name=...` | List S3 objects |
| GET | `/api/s3/read?name=...&key=...` | Read S3 object |

---

## GUI Walkthroughs

### Viewer: Search for errors in the last hour

1. Open http://localhost:9090
2. Select **"Error"** from the level dropdown
3. Type a search term or regex in the search bar
4. Click `.*` to enable regex mode
5. Results update live. Click **💾 Export** to download

### Collector: Watch live logs from multiple apps

1. Open http://localhost:8081
2. The **Live Feed** tab shows all incoming logs in real-time
3. Click a stream in the sidebar to filter
4. Click **⏸ Pause** to freeze the feed
5. Go to **⚙️ Configuration** to change rotation/compression settings

### Monitor: Set up a daily security scan

1. Open http://localhost:8082
2. Click **"+ New"**
3. Name: "Daily Security Audit"
4. Sources: `/var/log/auth.log` (one per line)
5. Patterns: `root.*login`, `sudo.*COMMAND`, `Failed password` (one per line)
6. Time Range: **Today**
7. Schedule: **Daily at** `07:00`
8. Add Telegram notification: paste bot token and chat ID
9. Click **"🧪 Test Notification"** to verify
10. Click **"💾 Save Rule"**
11. Click **"🧪 Test Now"** to see immediate results

---

## Example Scenarios

### 1. Development: All-in-one

```bash
LOG_MODE=both PORT=8080 LOG_SOURCES=./logs/*.log node dist-server/server.js
```
View logs + receive from apps on one port.

### 2. Production: Separate services

```bash
# Server A: Collector
LOG_MODE=collector PORT=8081 COLLECTOR_UDP_PORT=5140 \
  COLLECTOR_ROTATION_COMPRESSION=gzip \
  COLLECTOR_FORWARD_TARGETS='[{"type":"stream","targetStream":"all-logs"}]' \
  node dist-server/server.js

# Server B: Viewer
LOG_MODE=viewer PORT=9090 REMOTE_COLLECTOR_URL=http://serverA:8081 \
  node dist-server/server.js

# Server C: Monitor
LOG_MODE=monitor PORT=8082 MONITOR_SCAN_INTERVAL=30000 \
  node dist-server/server.js
```

### 3. Security monitoring with Telegram alerts

```bash
LOG_MODE=monitor PORT=8082 MONITOR_SCAN_INTERVAL=10000 \
  MONITOR_RULES='[{"id":"sec","name":"Security","enabled":true,"sources":["/var/log/auth.log"],"patterns":["root","sudo","Failed"],"patternLogic":"any","timeRange":{"type":"today"},"notifications":[{"type":"telegram","botToken":"YOUR_TOKEN","chatId":"YOUR_CHAT"}],"maxLinesInMessage":20,"cooldownMs":60000}]' \
  node dist-server/server.js
```

### 4. Log aggregation with S3 archive

```bash
LOG_MODE=both PORT=8080 \
  S3_ACCESS_KEY_ID=AKIA... S3_SECRET_ACCESS_KEY=... S3_REGION=eu-central-1 \
  S3_SOURCE_ARCHIVE=my-bucket:logs/ \
  COLLECTOR_FORWARD_TARGETS='[{"type":"file","path":"/archive/all.log"}]' \
  node dist-server/server.js
```

---

## Docker & CI/CD

### Build and run

```bash
docker build -t logmaster .

# Collector
docker run -d -p 8081:8080 -p 5140:5140/udp -e LOG_MODE=collector logmaster

# Viewer
docker run -d -p 9090:8080 -e LOG_MODE=viewer -e REMOTE_COLLECTOR_URL=http://collector:8080 logmaster

# Monitor
docker run -d -p 8082:8080 -e LOG_MODE=monitor -v /var/log:/logs:ro logmaster
```

### Docker Compose

```yaml
services:
  collector:
    build: .
    ports: ["8081:8080", "5140:5140/udp"]
    environment:
      LOG_MODE: collector
      COLLECTOR_UDP_PORT: "5140"
    volumes:
      - collector-data:/app/collector-data

  viewer:
    build: .
    ports: ["9090:8080"]
    environment:
      LOG_MODE: viewer
      REMOTE_COLLECTOR_URL: http://collector:8080
    depends_on: [collector]

  monitor:
    build: .
    ports: ["8082:8080"]
    environment:
      LOG_MODE: monitor
      MONITOR_SCAN_INTERVAL: "30000"
    volumes:
      - /var/log:/logs:ro

volumes:
  collector-data:
```

### CI/CD (Gitea Actions)

The `.gitea/workflows/ci.yaml` pipeline:
1. Installs dependencies
2. Compiles TypeScript
3. Generates sample logs
4. Starts all 3 services
5. Runs the full integration test suite (41 tests)
6. Builds Docker image
7. Pushes to Gitea container registry

To enable: add a Gitea Actions runner and set `REGISTRY_USER` / `REGISTRY_PASSWORD` secrets.

---

## Keyboard Shortcuts

| Key | Action |
|-----|--------|
| `Ctrl+F` | Search |
| `Ctrl+Shift+F` | Regex search |
| `Ctrl+T` | Toggle tail mode |
| `Ctrl+W` | Toggle word wrap |
| `Ctrl+G` | Go to line |
| `Ctrl+Shift+T` | Toggle theme |
| `Ctrl+=` / `Ctrl+-` | Zoom in/out |
| `Ctrl+0` | Reset zoom |
| `Home` / `End` | Scroll top/bottom |
| `Escape` | Clear search / close modals |

---

## Supported Log Formats

| Format | Extensions | Decompression |
|--------|-----------|---------------|
| Plain text | `.log`, `.txt`, no extension | Direct read |
| Gzip | `.gz` | Node.js zlib (built-in) |
| Bzip2 | `.bz2` | `bzcat` CLI |
| XZ | `.xz` | `xzcat` CLI |
| Zstandard | `.zst` | `zstdcat` CLI |

---

## License

MIT