# 📡 Log Collector — User Guide

## Overview

The Log Collector receives logs from applications via HTTP and UDP, stores them in named streams, rotates log files based on size/time, and optionally forwards them to other systems (databases, other collectors, files).

---

## Starting the Collector

### Option 1: Environment File
```bash
cp .env.collector .env
node dist-server/server.js
```

### Option 2: Inline
```bash
LOG_MODE=collector PORT=8081 COLLECTOR_UDP_PORT=5140 node dist-server/server.js
```

### Option 3: Docker
```bash
docker run -d -p 8081:8080 -p 5140:5140/udp \
  -e LOG_MODE=collector \
  -e COLLECTOR_UDP_PORT=5140 \
  -e COLLECTOR_ROTATION_COMPRESSION=gzip \
  -v ./collector-data:/app/collector-data \
  logviewer
```

### Option 4: Config File
```json
{
  "mode": "collector",
  "port": 8081,
  "collector": {
    "enabled": true,
    "storageDir": "./collector-data",
    "maxLinesPerStream": 100000,
    "maxStreams": 50,
    "persistToDisk": true,
    "udp": { "enabled": true, "port": 5140 },
    "rotation": {
      "maxSizeBytes": 5242880,
      "maxAgeMs": 600000,
      "compression": "gzip",
      "fileExtension": ".log"
    },
    "forwardTargets": []
  }
}
```

---

## Sending Logs to the Collector

### HTTP POST (Plain Text)

Each line in the body becomes one log entry:
```bash
curl -X POST http://localhost:8081/collect/my-app \
  -d "2026-04-30T10:00:01Z INFO Application started"
```

Multiple lines:
```bash
curl -X POST http://localhost:8081/collect/my-app -d "line 1
line 2
line 3"
```

### HTTP POST (JSON)

Send an array of strings or objects:
```bash
curl -X POST http://localhost:8081/collect/my-app/json \
  -H "Content-Type: application/json" \
  -d '["INFO App started", "ERROR Crash!", {"ts":"2026-04-30","level":"WARN","msg":"low memory"}]'
```

### UDP

Format: `streamId|logline` — the part before `|` is the stream name:
```bash
echo "my-app|2026-04-30 ERROR database timeout" | nc -u localhost 5140
```

Without stream ID (auto-named by sender IP):
```bash
echo "2026-04-30 INFO hello from UDP" | nc -u localhost 5140
```

### From Applications

**Node.js:**
```javascript
const http = require('http');
function log(stream, msg) {
  const req = http.request({ hostname:'localhost', port:8081, path:`/collect/${stream}`, method:'POST' });
  req.write(`${new Date().toISOString()} ${msg}`);
  req.end();
}
log('my-app', 'INFO Server started on port 3000');
```

**Python:**
```python
import requests, datetime
def log(stream, msg):
    ts = datetime.datetime.now().isoformat()
    requests.post(f'http://localhost:8081/collect/{stream}', data=f'{ts} {msg}')

log('my-app', 'INFO Processing complete')
```

**Bash (continuous pipe):**
```bash
tail -f /var/log/myapp.log | while read line; do
  curl -s -X POST http://localhost:8081/collect/myapp -d "$line"
done
```

### Test Script (included)

Sends random logs every 500ms:
```bash
node send-test-logs.js <stream-name> <collector-url>
node send-test-logs.js demo-app http://localhost:8081
```

---

## GUI Guide

### Layout

```
┌─────────────────────────────────────────────────────────────┐
│  📡 Log Collector [ACTIVE]                    Uptime: 5m 32s│
├──────────────┬──────────────────────────────────────────────┤
│              │  [🔴 Live Feed] [⚙️ Configuration] [🔀 Fwd]  │
│  📊 Overview │                                              │
│  ┌─────────┐ │  Live Feed:                                  │
│  │ 3       │ │  [demo-app] 10:00:01 INFO App started        │
│  │ Streams │ │  [demo-app] 10:00:02 ERROR DB timeout  ← red │
│  ├─────────┤ │  [web-srv] 10:00:03 WARN Slow request ← orange
│  │ 1.2k    │ │  [demo-app] 10:00:04 INFO Request OK         │
│  │ Lines   │ │                                              │
│  ├─────────┤ │  New lines appear with animation             │
│  │ 120     │ │                                              │
│  │ /min    │ │                                              │
│  └─────────┘ │                                              │
│              │                                              │
│  📡 Streams  │                                              │
│  ┌──────────┐│                                              │
│  │ demo-app ││                                              │
│  │ 450 lines││                                              │
│  │ 2s ago   ││                                              │
│  ├──────────┤│                                              │
│  │ web-srv  ││                                              │
│  │ 120 lines││                                              │
│  └──────────┘│                                              │
│              │                                              │
│  🔗 Endpoints                                               │
│  POST /collect/<id>                                         │
│  UDP :5140   │                                              │
└──────────────┴──────────────────────────────────────────────┘
```

### Tabs

**🔴 Live Feed**
- Shows incoming log lines in real-time
- Color-coded by level (red=ERROR, orange=WARN, blue=INFO)
- New lines animate in from the left
- Click a stream in the sidebar to filter the feed
- Pause/Resume button to freeze the feed
- Clear button to reset

**⚙️ Configuration**
- Edit all collector settings via the GUI
- Changes are saved to `logviewer.config.json`
- Settings: storage dir, max streams, max lines, UDP port, rotation size/age/compression

**🔀 Forwarding**
- View active forward targets
- Add new targets (HTTP, File, UDP, Console)
- Configure batch size and flush interval

---

## Configuration Reference

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `LOG_MODE` | `both` | Set to `collector` |
| `PORT` | `8080` | HTTP port |
| `COLLECTOR_UDP_ENABLED` | `true` | Enable/disable UDP receiver |
| `COLLECTOR_UDP_PORT` | `5140` | UDP listen port |
| `COLLECTOR_STORAGE_DIR` | `./collector-data` | Where to store log files |
| `COLLECTOR_MAX_LINES` | `100000` | Max lines kept in memory per stream |
| `COLLECTOR_MAX_STREAMS` | `50` | Max number of streams |
| `COLLECTOR_PERSIST` | `true` | Write logs to disk |
| `COLLECTOR_ROTATION_MAX_SIZE` | `5242880` | Rotate after N bytes (default 5MB) |
| `COLLECTOR_ROTATION_MAX_AGE` | `600000` | Rotate after N ms (default 10min) |
| `COLLECTOR_ROTATION_COMPRESSION` | `gzip` | `none`, `gzip`, `bzip2`, `xz`, `zstd` |
| `COLLECTOR_ROTATION_EXTENSION` | `.log` | File extension for active log |
| `COLLECTOR_FORWARD_TARGETS` | `[]` | JSON array of forward targets |

### Log Rotation

Rotation triggers on whichever condition is met **first**:

| Condition | Variable | Example |
|-----------|----------|---------|
| File size exceeds limit | `COLLECTOR_ROTATION_MAX_SIZE=10485760` | Rotate at 10MB |
| File age exceeds limit | `COLLECTOR_ROTATION_MAX_AGE=3600000` | Rotate every hour |

Rotated files are named with timestamp and compressed:
```
collector-data/
├── my-app.log                              ← current (active)
├── my-app_2026-04-30T10-00-00-000Z.log.gz  ← rotated + gzipped
├── my-app_2026-04-29T22-00-00-000Z.log.gz
└── _streams.json                           ← metadata
```

### Forward Targets

Forward targets receive copies of all ingested logs. Configure via environment or GUI.

**HTTP Target** — POST logs to another service:
```json
{"type":"http","url":"http://elasticsearch:9200/_bulk","format":"json","batchSize":200,"flushIntervalMs":5000}
```

**File Target** — Append to a file:
```json
{"type":"file","path":"/backup/all-logs.txt"}
```

**UDP Target** — Forward via UDP (e.g., to syslog):
```json
{"type":"udp","host":"syslog.internal","port":514}
```

**Stream Target** — Forward to another stream on the same collector:
```json
{"type":"stream","targetStream":"all-logs"}
```
This copies all ingested lines from any stream into the `all-logs` stream. Lines are prefixed with `[fwd:source-stream]` to identify the origin. Useful for creating an aggregated "all logs" stream.

**Console Target** — Print to stdout (for debugging):
```json
{"type":"console"}
```

**Multiple targets:**
```env
COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://elk:9200/_bulk","format":"json","batchSize":100},{"type":"file","path":"/backup/all.log"},{"type":"console"}]
```

---

## Example Configurations

### Minimal Collector
```env
LOG_MODE=collector
PORT=8081
```

### Production Collector (large scale)
```env
LOG_MODE=collector
PORT=8081
COLLECTOR_UDP_PORT=5140
COLLECTOR_STORAGE_DIR=/data/logs
COLLECTOR_MAX_LINES=500000
COLLECTOR_MAX_STREAMS=200
COLLECTOR_ROTATION_MAX_SIZE=52428800
COLLECTOR_ROTATION_MAX_AGE=3600000
COLLECTOR_ROTATION_COMPRESSION=zstd
COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://elasticsearch:9200/logs/_bulk","format":"json","batchSize":500,"flushIntervalMs":3000}]
```

### Collector with multiple forward targets
```env
LOG_MODE=collector
PORT=8081
COLLECTOR_FORWARD_TARGETS=[{"type":"http","url":"http://loki:3100/loki/api/v1/push","format":"json","batchSize":100},{"type":"file","path":"/archive/all-logs.txt"},{"type":"udp","host":"siem.internal","port":514}]
```

### Collector with stream aggregation (self-forward)
```env
LOG_MODE=collector
PORT=8081
COLLECTOR_FORWARD_TARGETS=[{"type":"stream","targetStream":"all-logs"},{"type":"console"}]
```
Every log sent to any stream (e.g., `web-app`, `auth-service`) is also copied into the `all-logs` stream. Lines are prefixed with `[fwd:web-app]` etc. so you can still tell where they came from.

### Docker Compose: Collector cluster
```yaml
services:
  collector-1:
    image: logviewer
    ports: ["8081:8080", "5140:5140/udp"]
    environment:
      LOG_MODE: collector
      COLLECTOR_UDP_PORT: "5140"
      COLLECTOR_ROTATION_COMPRESSION: gzip
      COLLECTOR_FORWARD_TARGETS: '[{"type":"http","url":"http://central:8080/collect/region-eu","format":"json"}]'
    volumes:
      - ./data-eu:/app/collector-data

  collector-2:
    image: logviewer
    ports: ["8082:8080", "5141:5140/udp"]
    environment:
      LOG_MODE: collector
      COLLECTOR_UDP_PORT: "5140"
      COLLECTOR_FORWARD_TARGETS: '[{"type":"http","url":"http://central:8080/collect/region-us","format":"json"}]'
    volumes:
      - ./data-us:/app/collector-data

  central:
    image: logviewer
    ports: ["9090:8080"]
    environment:
      LOG_MODE: both
```

---

## API Reference

| Method | Path | Description |
|--------|------|-------------|
| POST | `/collect/:streamId` | Ingest plain text (one line per line) |
| POST | `/collect/:streamId/json` | Ingest JSON array |
| GET | `/collector/streams` | List all streams |
| GET | `/collector/stream/:id?tail=N` | Read stream (optional: last N lines) |
| GET | `/collector/config` | Current collector config |
| DELETE | `/collector/stream/:id` | Delete a stream |

### Stream Auto-Creation

Streams are created automatically when you first send logs to them. No pre-configuration needed. Just POST to `/collect/any-name-you-want` and the stream exists.

### Stream Naming

Stream IDs support: `a-z`, `A-Z`, `0-9`, `-`, `_`, `.`

Examples: `my-app`, `web.server.prod`, `auth_service`, `node-1.api`