211 lines
7.0 KiB
Markdown
211 lines
7.0 KiB
Markdown
# DMARC Sentinel
|
|
|
|
DMARC Sentinel is a self-hosted Docker application for monitoring DMARC aggregate reports. It reads report emails from configured IMAP folders, extracts XML reports, stores normalized telemetry in SQLite, runs deterministic security and deliverability analysis, generates alerts, and uses an LLM to turn already-derived facts into human-readable summaries and recommendations.
|
|
|
|
Detection is deterministic. The LLM is mandatory for explanations, daily summaries, weekly summaries, and dashboard text, but it is never the source of truth for whether an alert exists or how severe it is.
|
|
|
|
## What It Does
|
|
|
|
- Connects to one or more IMAP inboxes and reads only the configured DMARC folder by default.
|
|
- Supports `.xml`, `.xml.gz`, `.gz`, and `.zip` report attachments.
|
|
- Parses DMARC aggregate XML into reports, records, and authentication results.
|
|
- Deduplicates reports by raw XML SHA256.
|
|
- Classifies known senders using CIDR allowlists, DKIM auth domains, SPF auth domains, and aligned DKIM evidence.
|
|
- Creates deterministic alerts for unknown failures, known sender failures, quarantine/reject dispositions, new sources, high failure rates, missing reporters, and first-seen policies/reporters.
|
|
- Uses OpenAI to produce alert explanations and daily/weekly operational summaries from sanitized JSON facts.
|
|
- Exposes a server-rendered FastAPI dashboard and gethomepage Custom API widget endpoints.
|
|
- Sends email alerts and digests when SMTP settings are configured.
|
|
|
|
## First Run
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
cp config/config.example.yml config/config.yml
|
|
nano .env
|
|
nano config/config.yml
|
|
docker compose up -d --build
|
|
```
|
|
|
|
Then process the existing mailbox backlog:
|
|
|
|
```bash
|
|
docker compose exec dmarc-sentinel python -m app.cli backlog --inbox tukutoi --folder DMARC
|
|
```
|
|
|
|
Visit the app through your NPM reverse proxy:
|
|
|
|
```text
|
|
https://sentinel.tukutoi.com
|
|
```
|
|
|
|
## mailcow / SOGo Expectation
|
|
|
|
No installation is required on the mailcow server. The app logs into the existing mailbox over IMAP.
|
|
|
|
Expected mailbox setup:
|
|
|
|
```yaml
|
|
mail_account: hello@tukutoi.com
|
|
catch_all: true
|
|
reports_recipient: dmarcreports@tukutoi.com
|
|
imap_folder: DMARC
|
|
primary_domain: tukutoi.com
|
|
```
|
|
|
|
Move or filter DMARC aggregate report messages into the `DMARC` folder in SOGo/mailcow. DMARC Sentinel reads that folder unless a CLI/API backlog option overrides it.
|
|
|
|
## Docker Compose
|
|
|
|
The compose file binds the dashboard to `127.0.0.1:8000` for local HITL testing and also attaches to the external NPM proxy network with a static container IP:
|
|
|
|
```yaml
|
|
networks:
|
|
npm_proxy:
|
|
external: true
|
|
```
|
|
|
|
NPM should proxy to:
|
|
|
|
```text
|
|
http://192.168.99.18:8000
|
|
```
|
|
|
|
The app listens inside the container on `0.0.0.0:8000`; on a local Docker host it is also available at:
|
|
|
|
```text
|
|
http://127.0.0.1:8000
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Runtime config is read from `config/config.yml`; use `config/config.example.yml` as the template.
|
|
|
|
Important sections:
|
|
|
|
- `app`: base URL, timezone, poll interval, database URL, log level, attachment limits.
|
|
- `security`: Basic Auth and homepage bearer-token settings.
|
|
- `llm`: OpenAI model, retry, timeout, and data-sending controls. Raw XML and raw email are disabled by default.
|
|
- `inboxes`: IMAP host, folder, recipient, processed/failed folder behavior, and env var names for credentials.
|
|
- `known_senders`: deterministic sender classification rules per domain.
|
|
- `alerts`: SMTP env var names and deterministic thresholds.
|
|
|
|
Secrets live in `.env`, not in `config.yml`.
|
|
|
|
`OPENAI_API_KEY` is required when `llm.provider` is `openai`. The app only bypasses this when `DMARC_SENTINEL_ALLOW_NO_LLM_FOR_TESTS=true`, which is intended for tests.
|
|
|
|
Adding another inbox or monitored domain should only require adding entries to `inboxes` and `known_senders`.
|
|
|
|
## Backlog Command
|
|
|
|
```bash
|
|
docker compose exec dmarc-sentinel python -m app.cli backlog --inbox tukutoi
|
|
```
|
|
|
|
Options:
|
|
|
|
```text
|
|
--folder DMARC
|
|
--since YYYY-MM-DD
|
|
--before YYYY-MM-DD
|
|
--limit 500
|
|
--dry-run
|
|
--reprocess
|
|
--mark-seen
|
|
```
|
|
|
|
Backlog mode scans all matching messages, skips already imported XML hashes unless `--reprocess` is passed, and does not modify messages unless configured or explicitly flagged.
|
|
|
|
## gethomepage Widget
|
|
|
|
Endpoint:
|
|
|
|
```http
|
|
GET /api/homepage
|
|
Authorization: Bearer YOUR_HOMEPAGE_API_TOKEN
|
|
```
|
|
|
|
Example gethomepage config:
|
|
|
|
```yaml
|
|
- Monitoring:
|
|
- DMARC Sentinel:
|
|
href: https://sentinel.tukutoi.com
|
|
description: DMARC monitoring
|
|
widget:
|
|
type: customapi
|
|
url: https://sentinel.tukutoi.com/api/homepage
|
|
headers:
|
|
Authorization: Bearer YOUR_HOMEPAGE_API_TOKEN
|
|
mappings:
|
|
- field: status
|
|
label: Status
|
|
- field: dmarc_pass_rate
|
|
label: Pass
|
|
- field: critical_alerts
|
|
label: Critical
|
|
- field: warnings
|
|
label: Warnings
|
|
- field: summary
|
|
label: Summary
|
|
```
|
|
|
|
Domain-specific endpoint:
|
|
|
|
```http
|
|
GET /api/homepage/tukutoi.com
|
|
```
|
|
|
|
## Dashboard
|
|
|
|
The dashboard is protected with Basic Auth using `DASHBOARD_USERNAME` and `DASHBOARD_PASSWORD`. It includes:
|
|
|
|
- `/` overview with monitored domains, daily volume, pass rate, alerts, unknown sources, last check, and latest LLM daily summary.
|
|
- `/domains/{domain}` with trends, top sources, known vs unknown sender data, reports, alerts, and LLM summary.
|
|
- `/reports/{report_id}` with normalized report evidence and record table.
|
|
- `/alerts` with acknowledge, resolve, and reopen actions.
|
|
- `/inboxes` with status and manual process/backlog buttons.
|
|
|
|
## HTTP API
|
|
|
|
Implemented endpoints:
|
|
|
|
```text
|
|
GET /health
|
|
GET /api/homepage
|
|
GET /api/homepage/{domain}
|
|
GET /api/domains
|
|
GET /api/domains/{domain}/summary
|
|
GET /api/domains/{domain}/reports
|
|
GET /api/domains/{domain}/sources
|
|
GET /api/reports/{id}
|
|
GET /api/alerts
|
|
POST /api/alerts/{id}/ack
|
|
POST /api/alerts/{id}/resolve
|
|
POST /api/alerts/{id}/reopen
|
|
POST /api/admin/process-now
|
|
POST /api/admin/backlog
|
|
```
|
|
|
|
`/health` is public. Homepage API routes use bearer auth when enabled. Dashboard and admin/API management routes use Basic Auth.
|
|
|
|
## Troubleshooting
|
|
|
|
- `OPENAI_API_KEY is required`: set it in `.env`. Do not use the test bypass in production.
|
|
- IMAP folder errors: confirm the folder is exactly named `DMARC` and exists for `hello@tukutoi.com`.
|
|
- No reports imported: check that messages contain valid DMARC aggregate XML attachments and that they are in the configured folder.
|
|
- Duplicate reports skipped: the raw XML SHA256 has already been imported.
|
|
- NPM cannot connect: confirm the `npm_proxy` Docker network exists and the container has `192.168.99.18`.
|
|
- Dashboard login fails: confirm `DASHBOARD_USERNAME` and `DASHBOARD_PASSWORD` in `.env`.
|
|
- Homepage widget returns 401: confirm `HOMEPAGE_API_TOKEN` and the `Authorization: Bearer ...` header.
|
|
- Email alerts do not send: verify SMTP host, port, username, password, sender, and recipient env vars.
|
|
|
|
## Development
|
|
|
|
Run tests:
|
|
|
|
```bash
|
|
DMARC_SENTINEL_ALLOW_NO_LLM_FOR_TESTS=true pytest
|
|
```
|
|
|
|
The XML parser uses `defusedxml`, ZIP extraction rejects path traversal and nested archives, and decompressed attachment sizes are capped by config.
|