Nautilus/ledgerdock
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: complete repository technical documentation refresh

Browse Source
This commit is contained in:
Beda Schmid
2026-02-21 11:27:44 -03:00
parent 17dccbbe20
commit 6501841426
6 changed files with 454 additions and 88 deletions
Download Patch File Download Diff File Expand all files Collapse all files

161
README.md
View File

@@ -1,118 +1,121 @@
# DMS # DMS
DMS is a single-user document management system with: DMS is a single-user document management system for ingesting, processing, organizing, and searching files.
- drag-and-drop upload anywhere in the UI
- file and folder upload Core capabilities:
- document processing and indexing (PDF, text, OpenAI handwriting/image transcription, DOCX, XLSX, ZIP extraction) - drag-and-drop upload from anywhere in the UI
- fallback handling for unsupported formats - file and folder upload with path preservation
- original file preservation and download - asynchronous processing with OCR and extraction for PDF, images, DOCX, XLSX, TXT, and ZIP
- metadata-based and full-text search - metadata and full-text search
- learning-based routing suggestions - routing suggestions based on learned decisions
- original file download and extracted markdown export
## Stack
- Backend: FastAPI + SQLAlchemy + RQ worker (`backend/`)
- Frontend: React + Vite + TypeScript (`frontend/`)
- Infrastructure: Postgres, Redis, Typesense (`docker-compose.yml`)
## Requirements ## Requirements
- Docker Engine with Docker Compose plugin - Docker Engine
- Docker Compose plugin
- Internet access for the first image build - Internet access for the first image build
## Install And Run With Docker Compose ## Quick Start
1. Open a terminal in this repository root. 1. Start the full stack from repository root:
2. Start the full stack:
```bash ```bash
docker compose up --build -d docker compose up --build -d
``` ```
3. Open the applications: 2. Open services:
- Frontend: `http://localhost:5173` - Frontend: `http://localhost:5173`
- Backend API docs: `http://localhost:8000/docs` - Backend OpenAPI docs: `http://localhost:8000/docs`
- Health check: `http://localhost:8000/api/v1/health` - Health endpoint: `http://localhost:8000/api/v1/health`
## Setup 3. Stop when done:
1. Open the frontend and upload files or folders.
2. Set default destination path and tags before upload if needed.
3. Configure handwriting transcription settings in the UI:
- OpenAI compatible base URL
- model (default: `gpt-4.1-mini`)
- API key and timeout
4. Open a document in the details panel, adjust destination and tags, then save.
5. Keep `Learn from this routing decision` enabled to train future routing suggestions.
## Data Persistence On Host
All runtime data is stored on the host using bind mounts.
Default host location:
- `./data/postgres`
- `./data/redis`
- `./data/storage`
To persist under another host directory, set `DCM_DATA_DIR`:
```bash
DCM_DATA_DIR=/data docker compose up --build -d
```
This will place runtime data under `/data` on the host.
## Common Commands
Start:
```bash
docker compose up --build -d
```
Stop:
```bash ```bash
docker compose down docker compose down
``` ```
View logs: ## Common Commands
Start services:
```bash
docker compose up --build -d
```
Stop services:
```bash
docker compose down
```
Stream logs:
```bash ```bash
docker compose logs -f docker compose logs -f
``` ```
Rebuild a clean stack while keeping persisted data: Rebuild services:
```bash ```bash
docker compose down docker compose down
docker compose up --build -d docker compose up --build -d
``` ```
Reset all persisted runtime data: Reset runtime data (destructive, removes named volumes):
```bash ```bash
docker compose down docker compose down -v
rm -rf ./data
``` ```
## Handwriting Transcription Notes ## Data Persistence
- Handwriting and image transcription uses an OpenAI compatible vision endpoint. Runtime state is persisted in Docker named volumes declared in `docker-compose.yml`:
- Before transcription, images are normalized: - `db-data`
- EXIF rotation is corrected - `redis-data`
- long side is resized to a maximum of 2000px - `dcm-storage`
- image is sent as a base64 data URL payload - `typesense-data`
- Handwriting provider settings are persisted in host storage and survive container restarts.
## API Overview The application settings file is stored under the storage volume at `/data/storage/settings.json` inside containers.
GET endpoints: ## Configuration Notes
- `GET /api/v1/health`
- `GET /api/v1/documents`
- `GET /api/v1/documents/{document_id}`
- `GET /api/v1/documents/{document_id}/preview`
- `GET /api/v1/documents/{document_id}/download`
- `GET /api/v1/documents/tags`
- `GET /api/v1/search?query=...`
- `GET /api/v1/settings`
Additional endpoints used by the UI: - API and worker runtime environment values are configured in `docker-compose.yml` (`DATABASE_URL`, `REDIS_URL`, `STORAGE_ROOT`, `PUBLIC_BASE_URL`, `CORS_ORIGINS`, `TYPESENSE_*`).
- `POST /api/v1/documents/upload` - Frontend API target is controlled by `VITE_API_BASE` in the `frontend` service.
- `PATCH /api/v1/documents/{document_id}` - Handwriting, provider, routing, summary, display, and upload defaults are managed through the settings UI and persisted by the backend settings service.
- `POST /api/v1/documents/{document_id}/reprocess`
- `PATCH /api/v1/settings/handwriting` ## Manual Validation Checklist
After changes, verify:
- `GET /api/v1/health` returns `{"status":"ok"}`
- upload and processing complete successfully
- search returns expected results
- preview or download works for uploaded documents
- `docker compose logs -f` shows no API or worker failures
## API Surface Summary
Base prefix: `/api/v1`
- Health: `/health`
- Documents: `/documents` (listing, upload, metadata update, lifecycle actions, download and preview, markdown export)
- Search: `/search`
- Processing logs: `/processing/logs`
- Settings: `/settings` and `/settings/handwriting`
See `doc/api-contract.md` for the complete endpoint contract.
## Technical Documentation
- `doc/README.md` - technical documentation index
- `doc/architecture-overview.md` - service and runtime architecture
- `doc/api-contract.md` - HTTP endpoint contract and payload model map
- `doc/data-model-reference.md` - database model reference
- `doc/operations-and-configuration.md` - operations runbook and configuration reference
- `doc/frontend-design-foundation.md` - frontend design system and UI rules

21
doc/README.md
View File

@@ -1,15 +1,18 @@
# Documentation # Documentation
This is the documentation entrypoint for DMS. This directory contains technical documentation for DMS.
## Available Documents ## Core References
- Project setup and operations: `../README.md` - `../README.md` - project overview, setup, and quick operations
- Frontend visual system and compact UI rules: `frontend-design-foundation.md` - `architecture-overview.md` - backend, frontend, and infrastructure architecture
- Handwriting style implementation plan: `../PLAN.md` - `api-contract.md` - API endpoint contract grouped by route module
- `data-model-reference.md` - database entity definitions and lifecycle states
- `operations-and-configuration.md` - runtime operations, ports, volumes, and configuration values
- `frontend-design-foundation.md` - frontend visual system, tokens, and UI implementation rules
## Planned Additions ## Documentation Rules
- Architecture overview - Keep this file as the documentation index and add new technical documents here.
- Data model reference - Update referenced documents whenever behavior, routes, models, or runtime configuration change.
- API contract details - Prefer concise, implementation-backed descriptions with explicit paths to source modules.

130
doc/api-contract.md Normal file
View File

@@ -0,0 +1,130 @@
# API Contract
Base URL prefix: `/api/v1`
Primary implementation modules:
- `backend/app/api/router.py`
- `backend/app/api/routes_health.py`
- `backend/app/api/routes_documents.py`
- `backend/app/api/routes_search.py`
- `backend/app/api/routes_processing_logs.py`
- `backend/app/api/routes_settings.py`
## Health
- `GET /health`
- Purpose: liveness check
- Response: `{ "status": "ok" }`
## Documents
### Collection and metadata helpers
- `GET /documents`
- Query: `offset`, `limit`, `include_trashed`, `only_trashed`, `path_prefix`, `path_filter`, `tag_filter`, `type_filter`, `processed_from`, `processed_to`
- Response model: `DocumentsListResponse`
- `GET /documents/tags`
- Query: `include_trashed`
- Response: `{ "tags": string[] }`
- `GET /documents/paths`
- Query: `include_trashed`
- Response: `{ "paths": string[] }`
- `GET /documents/types`
- Query: `include_trashed`
- Response: `{ "types": string[] }`
- `POST /documents/content-md/export`
- Body model: `ContentExportRequest`
- Response: ZIP stream containing one markdown file per matched document
### Per-document operations
- `GET /documents/{document_id}`
- Response model: `DocumentDetailResponse`
- `GET /documents/{document_id}/download`
- Response: original file bytes
- `GET /documents/{document_id}/preview`
- Response: inline preview stream where browser-supported
- `GET /documents/{document_id}/thumbnail`
- Response: generated thumbnail image when available
- `GET /documents/{document_id}/content-md`
- Response: extracted markdown content for one document
- `PATCH /documents/{document_id}`
- Body model: `DocumentUpdateRequest`
- Response model: `DocumentResponse`
- `POST /documents/{document_id}/trash`
- Response model: `DocumentResponse`
- `POST /documents/{document_id}/restore`
- Response model: `DocumentResponse`
- `DELETE /documents/{document_id}`
- Behavior: permanent delete, requires document to be trashed first
- Response: deletion counters
- `POST /documents/{document_id}/reprocess`
- Response model: `DocumentResponse`
- Behavior: requeues asynchronous processing task
### Upload
- `POST /documents/upload`
- Multipart form fields:
- `files[]` (required)
- `relative_paths[]` (optional)
- `logical_path` (optional, defaults to `Inbox`)
- `tags` (optional CSV)
- `conflict_mode` (`ask`, `replace`, `duplicate`)
- Response model: `UploadResponse`
- Behavior:
- `ask`: returns `conflicts` if duplicate checksum is detected
- `replace`: creates new document linked to replaced document id
- `duplicate`: creates additional document record
## Search
- `GET /search`
- Query: `query` (min length 2), `offset`, `limit`, `include_trashed`, `only_trashed`, `path_filter`, `tag_filter`, `type_filter`, `processed_from`, `processed_to`
- Response model: `SearchResponse`
- Behavior: PostgreSQL full-text and metadata ranking
## Processing Logs
- `GET /processing/logs`
- Query: `offset`, `limit`, `document_id`
- Response model: `ProcessingLogListResponse`
- `POST /processing/logs/trim`
- Query: `keep_document_sessions`, `keep_unbound_entries`
- Response: trim counters
- `POST /processing/logs/clear`
- Response: clear counters
## Settings
- `GET /settings`
- Response model: `AppSettingsResponse`
- `PATCH /settings`
- Body model: `AppSettingsUpdateRequest`
- Response model: `AppSettingsResponse`
- `POST /settings/reset`
- Response model: `AppSettingsResponse`
- `PATCH /settings/handwriting`
- Body model: `HandwritingSettingsUpdateRequest`
- Response model: `AppSettingsResponse`
- `GET /settings/handwriting`
- Response model: `HandwritingSettingsResponse`
## Schema Families
Document schemas in `backend/app/schemas/documents.py`:
- `DocumentResponse`
- `DocumentDetailResponse`
- `DocumentsListResponse`
- `UploadConflict`
- `UploadResponse`
- `DocumentUpdateRequest`
- `SearchResponse`
- `ContentExportRequest`
Processing log schemas in `backend/app/schemas/processing_logs.py`:
- `ProcessingLogEntryResponse`
- `ProcessingLogListResponse`
Settings schemas in `backend/app/schemas/settings.py`:
- Provider, task, upload-default, display, predefined paths or tags, handwriting-style, and legacy handwriting models grouped under `AppSettingsResponse` and `AppSettingsUpdateRequest`.

66
doc/architecture-overview.md Normal file
View File

@@ -0,0 +1,66 @@
# Architecture Overview
## System Topology
DMS runs as a multi-service application defined in `docker-compose.yml`:
- `frontend` serves the React UI on port `5173`
- `api` serves FastAPI on port `8000`
- `worker` executes asynchronous extraction and indexing jobs
- `db` provides PostgreSQL persistence on port `5432`
- `redis` backs queueing on port `6379`
- `typesense` stores search index and vector-adjacent metadata on port `8108`
## Backend Architecture
Backend source root: `backend/app/`
Main boundaries:
- `api/` route handlers and HTTP contract
- `services/` domain logic (storage, extraction, routing, settings, processing logs, Typesense)
- `db/` SQLAlchemy base, engine, and session lifecycle
- `models/` persistence entities (`Document`, `ProcessingLogEntry`)
- `schemas/` Pydantic response and request schemas
- `worker/` RQ queue integration and background processing tasks
Application bootstrap in `backend/app/main.py`:
- mounts routers under `/api/v1`
- configures CORS from settings
- initializes storage, settings, database schema, and Typesense collection on startup
## Processing Lifecycle
1. Upload starts at `POST /api/v1/documents/upload`.
2. API stores file bytes and inserts document rows with status `queued`.
3. API enqueues `app.worker.tasks.process_document_task` into Redis.
4. Worker extracts content and metadata, handles ZIP expansion, runs OCR and routing suggestions, and writes processing logs.
5. Worker updates database fields, document status, and search index entries.
6. UI polls for documents and processing logs to reflect progress.
## Frontend Architecture
Frontend source root: `frontend/src/`
Core structure:
- `App.tsx` orchestrates screen switching, state, polling, and action flows
- `components/` contains upload, filter, grid, viewer, modal, settings, and log panel modules
- `lib/api.ts` centralizes API client calls
- `types.ts` defines typed API contracts used by components
- `design-foundation.css` and `styles.css` define design tokens and global/component styling
Main user flows:
- Upload and conflict resolution
- Search and filtered document browsing
- Metadata editing and lifecycle actions (trash, restore, delete, reprocess)
- Settings management for providers, tasks, and UI defaults
- Processing log review
## Persistence and State
Persistent data:
- PostgreSQL stores document metadata and processing logs
- Docker volume-backed storage keeps original files, previews, and settings JSON
- Typesense stores indexed search representations
Transient runtime state:
- Redis queues processing tasks and worker execution state
- frontend local component state drives active filters, selection, and modal flows

53
doc/data-model-reference.md Normal file
View File

@@ -0,0 +1,53 @@
# Data Model Reference
Primary SQLAlchemy models are defined in `backend/app/models/`.
## documents
Model: `Document` in `backend/app/models/document.py`
Purpose:
- Stores source file identity, storage location, extracted content, lifecycle status, and classification metadata.
Core fields:
- Identity and source: `id`, `original_filename`, `source_relative_path`, `stored_relative_path`
- File attributes: `mime_type`, `extension`, `sha256`, `size_bytes`
- Organization: `logical_path`, `suggested_path`, `tags`, `suggested_tags`
- Processing outputs: `extracted_text`, `image_text_type`, `handwriting_style_id`, `preview_available`
- Lifecycle and relations: `status`, `is_archive_member`, `archived_member_path`, `parent_document_id`, `replaces_document_id`
- Metadata and timestamps: `metadata_json`, `created_at`, `processed_at`, `updated_at`
Enum `DocumentStatus`:
- `queued`
- `processed`
- `unsupported`
- `error`
- `trashed`
Relationships:
- Self-referential `parent_document` relationship for archive extraction trees.
## processing_logs
Model: `ProcessingLogEntry` in `backend/app/models/processing_log.py`
Purpose:
- Stores timestamped pipeline events for upload, extraction, OCR, routing, indexing, and errors.
Core fields:
- Event identity and timing: `id`, `created_at`
- Event classification: `level`, `stage`, `event`
- Document linkage: `document_id`, `document_filename`
- Model context: `provider_id`, `model_name`
- Prompt or response traces: `prompt_text`, `response_text`
- Structured event payload: `payload_json`
Foreign keys:
- `document_id` references `documents.id` with `ON DELETE SET NULL`.
## Model Lifecycle Notes
- Upload inserts a `Document` row in `queued` state and enqueues background processing.
- Worker updates extraction results and final status (`processed`, `unsupported`, or `error`).
- Trash and restore operations toggle `status` while preserving source files until permanent delete.
- Permanent delete removes the document tree (including archive descendants) and associated stored files.

111
doc/operations-and-configuration.md Normal file
View File

@@ -0,0 +1,111 @@
# Operations And Configuration
## Runtime Services
`docker-compose.yml` defines the runtime stack:
- `db` (Postgres 16, port `5432`)
- `redis` (Redis 7, port `6379`)
- `typesense` (Typesense 29, port `8108`)
- `api` (FastAPI backend, port `8000`)
- `worker` (RQ background worker)
- `frontend` (Vite UI, port `5173`)
## Named Volumes
Persistent volumes:
- `db-data`
- `redis-data`
- `dcm-storage`
- `typesense-data`
Reset all persisted runtime data:
```bash
docker compose down -v
```
## Operational Commands
Start or rebuild stack:
```bash
docker compose up --build -d
```
Stop stack:
```bash
docker compose down
```
Tail logs:
```bash
docker compose logs -f
```
## Backend Configuration
Settings source:
- Runtime settings class: `backend/app/core/config.py`
- API settings persistence: `backend/app/services/app_settings.py`
Key environment variables used by `api` and `worker` in compose:
- `APP_ENV`
- `DATABASE_URL`
- `REDIS_URL`
- `STORAGE_ROOT`
- `PUBLIC_BASE_URL`
- `CORS_ORIGINS` (API service)
- `TYPESENSE_PROTOCOL`
- `TYPESENSE_HOST`
- `TYPESENSE_PORT`
- `TYPESENSE_API_KEY`
- `TYPESENSE_COLLECTION_NAME`
Selected defaults from `Settings` (`backend/app/core/config.py`):
- `upload_chunk_size = 4194304`
- `max_zip_members = 250`
- `max_zip_depth = 2`
- `max_text_length = 500000`
- `default_openai_model = "gpt-4.1-mini"`
- `default_openai_timeout_seconds = 45`
- `default_summary_model = "gpt-4.1-mini"`
- `default_routing_model = "gpt-4.1-mini"`
- `typesense_timeout_seconds = 120`
- `typesense_num_retries = 0`
## Frontend Configuration
Frontend runtime API target:
- `VITE_API_BASE` in `docker-compose.yml` frontend service
Frontend local commands:
```bash
cd frontend && npm run dev
cd frontend && npm run build
cd frontend && npm run preview
```
## Settings Persistence
Application-level settings managed from the UI are persisted by backend settings service:
- file path: `<STORAGE_ROOT>/settings.json`
- endpoints: `/api/v1/settings`, `/api/v1/settings/reset`, `/api/v1/settings/handwriting`
Settings include:
- upload defaults
- display options
- provider configuration
- OCR, summary, and routing task settings
- predefined paths and tags
- handwriting-style clustering settings
## Validation Checklist
After operational or configuration changes, verify:
- `GET /api/v1/health` is healthy
- frontend can list, upload, and search documents
- processing worker logs show successful task execution
- settings save or reset works and persists after restart