diff --git a/README.md b/README.md index 76bcd6b..a9bab21 100644 --- a/README.md +++ b/README.md @@ -1,118 +1,121 @@ # DMS -DMS is a single-user document management system with: -- drag-and-drop upload anywhere in the UI -- file and folder upload -- document processing and indexing (PDF, text, OpenAI handwriting/image transcription, DOCX, XLSX, ZIP extraction) -- fallback handling for unsupported formats -- original file preservation and download -- metadata-based and full-text search -- learning-based routing suggestions +DMS is a single-user document management system for ingesting, processing, organizing, and searching files. + +Core capabilities: +- drag-and-drop upload from anywhere in the UI +- file and folder upload with path preservation +- asynchronous processing with OCR and extraction for PDF, images, DOCX, XLSX, TXT, and ZIP +- metadata and full-text search +- routing suggestions based on learned decisions +- original file download and extracted markdown export + +## Stack + +- Backend: FastAPI + SQLAlchemy + RQ worker (`backend/`) +- Frontend: React + Vite + TypeScript (`frontend/`) +- Infrastructure: Postgres, Redis, Typesense (`docker-compose.yml`) ## Requirements -- Docker Engine with Docker Compose plugin +- Docker Engine +- Docker Compose plugin - Internet access for the first image build -## Install And Run With Docker Compose +## Quick Start -1. Open a terminal in this repository root. -2. Start the full stack: +1. Start the full stack from repository root: ```bash docker compose up --build -d ``` -3. Open the applications: +2. Open services: - Frontend: `http://localhost:5173` -- Backend API docs: `http://localhost:8000/docs` -- Health check: `http://localhost:8000/api/v1/health` +- Backend OpenAPI docs: `http://localhost:8000/docs` +- Health endpoint: `http://localhost:8000/api/v1/health` -## Setup - -1. Open the frontend and upload files or folders. -2. Set default destination path and tags before upload if needed. -3. Configure handwriting transcription settings in the UI: -- OpenAI compatible base URL -- model (default: `gpt-4.1-mini`) -- API key and timeout -4. Open a document in the details panel, adjust destination and tags, then save. -5. Keep `Learn from this routing decision` enabled to train future routing suggestions. - -## Data Persistence On Host - -All runtime data is stored on the host using bind mounts. - -Default host location: -- `./data/postgres` -- `./data/redis` -- `./data/storage` - -To persist under another host directory, set `DCM_DATA_DIR`: - -```bash -DCM_DATA_DIR=/data docker compose up --build -d -``` - -This will place runtime data under `/data` on the host. - -## Common Commands - -Start: - -```bash -docker compose up --build -d -``` - -Stop: +3. Stop when done: ```bash docker compose down ``` -View logs: +## Common Commands + +Start services: + +```bash +docker compose up --build -d +``` + +Stop services: + +```bash +docker compose down +``` + +Stream logs: ```bash docker compose logs -f ``` -Rebuild a clean stack while keeping persisted data: +Rebuild services: ```bash docker compose down docker compose up --build -d ``` -Reset all persisted runtime data: +Reset runtime data (destructive, removes named volumes): ```bash -docker compose down -rm -rf ./data +docker compose down -v ``` -## Handwriting Transcription Notes +## Data Persistence -- Handwriting and image transcription uses an OpenAI compatible vision endpoint. -- Before transcription, images are normalized: - - EXIF rotation is corrected - - long side is resized to a maximum of 2000px - - image is sent as a base64 data URL payload -- Handwriting provider settings are persisted in host storage and survive container restarts. +Runtime state is persisted in Docker named volumes declared in `docker-compose.yml`: +- `db-data` +- `redis-data` +- `dcm-storage` +- `typesense-data` -## API Overview +The application settings file is stored under the storage volume at `/data/storage/settings.json` inside containers. -GET endpoints: -- `GET /api/v1/health` -- `GET /api/v1/documents` -- `GET /api/v1/documents/{document_id}` -- `GET /api/v1/documents/{document_id}/preview` -- `GET /api/v1/documents/{document_id}/download` -- `GET /api/v1/documents/tags` -- `GET /api/v1/search?query=...` -- `GET /api/v1/settings` +## Configuration Notes -Additional endpoints used by the UI: -- `POST /api/v1/documents/upload` -- `PATCH /api/v1/documents/{document_id}` -- `POST /api/v1/documents/{document_id}/reprocess` -- `PATCH /api/v1/settings/handwriting` +- API and worker runtime environment values are configured in `docker-compose.yml` (`DATABASE_URL`, `REDIS_URL`, `STORAGE_ROOT`, `PUBLIC_BASE_URL`, `CORS_ORIGINS`, `TYPESENSE_*`). +- Frontend API target is controlled by `VITE_API_BASE` in the `frontend` service. +- Handwriting, provider, routing, summary, display, and upload defaults are managed through the settings UI and persisted by the backend settings service. + +## Manual Validation Checklist + +After changes, verify: +- `GET /api/v1/health` returns `{"status":"ok"}` +- upload and processing complete successfully +- search returns expected results +- preview or download works for uploaded documents +- `docker compose logs -f` shows no API or worker failures + +## API Surface Summary + +Base prefix: `/api/v1` + +- Health: `/health` +- Documents: `/documents` (listing, upload, metadata update, lifecycle actions, download and preview, markdown export) +- Search: `/search` +- Processing logs: `/processing/logs` +- Settings: `/settings` and `/settings/handwriting` + +See `doc/api-contract.md` for the complete endpoint contract. + +## Technical Documentation + +- `doc/README.md` - technical documentation index +- `doc/architecture-overview.md` - service and runtime architecture +- `doc/api-contract.md` - HTTP endpoint contract and payload model map +- `doc/data-model-reference.md` - database model reference +- `doc/operations-and-configuration.md` - operations runbook and configuration reference +- `doc/frontend-design-foundation.md` - frontend design system and UI rules diff --git a/doc/README.md b/doc/README.md index e1de414..379882c 100644 --- a/doc/README.md +++ b/doc/README.md @@ -1,15 +1,18 @@ # Documentation -This is the documentation entrypoint for DMS. +This directory contains technical documentation for DMS. -## Available Documents +## Core References -- Project setup and operations: `../README.md` -- Frontend visual system and compact UI rules: `frontend-design-foundation.md` -- Handwriting style implementation plan: `../PLAN.md` +- `../README.md` - project overview, setup, and quick operations +- `architecture-overview.md` - backend, frontend, and infrastructure architecture +- `api-contract.md` - API endpoint contract grouped by route module +- `data-model-reference.md` - database entity definitions and lifecycle states +- `operations-and-configuration.md` - runtime operations, ports, volumes, and configuration values +- `frontend-design-foundation.md` - frontend visual system, tokens, and UI implementation rules -## Planned Additions +## Documentation Rules -- Architecture overview -- Data model reference -- API contract details +- Keep this file as the documentation index and add new technical documents here. +- Update referenced documents whenever behavior, routes, models, or runtime configuration change. +- Prefer concise, implementation-backed descriptions with explicit paths to source modules. diff --git a/doc/api-contract.md b/doc/api-contract.md new file mode 100644 index 0000000..e596462 --- /dev/null +++ b/doc/api-contract.md @@ -0,0 +1,130 @@ +# API Contract + +Base URL prefix: `/api/v1` + +Primary implementation modules: +- `backend/app/api/router.py` +- `backend/app/api/routes_health.py` +- `backend/app/api/routes_documents.py` +- `backend/app/api/routes_search.py` +- `backend/app/api/routes_processing_logs.py` +- `backend/app/api/routes_settings.py` + +## Health + +- `GET /health` +- Purpose: liveness check +- Response: `{ "status": "ok" }` + +## Documents + +### Collection and metadata helpers + +- `GET /documents` + - Query: `offset`, `limit`, `include_trashed`, `only_trashed`, `path_prefix`, `path_filter`, `tag_filter`, `type_filter`, `processed_from`, `processed_to` + - Response model: `DocumentsListResponse` +- `GET /documents/tags` + - Query: `include_trashed` + - Response: `{ "tags": string[] }` +- `GET /documents/paths` + - Query: `include_trashed` + - Response: `{ "paths": string[] }` +- `GET /documents/types` + - Query: `include_trashed` + - Response: `{ "types": string[] }` +- `POST /documents/content-md/export` + - Body model: `ContentExportRequest` + - Response: ZIP stream containing one markdown file per matched document + +### Per-document operations + +- `GET /documents/{document_id}` + - Response model: `DocumentDetailResponse` +- `GET /documents/{document_id}/download` + - Response: original file bytes +- `GET /documents/{document_id}/preview` + - Response: inline preview stream where browser-supported +- `GET /documents/{document_id}/thumbnail` + - Response: generated thumbnail image when available +- `GET /documents/{document_id}/content-md` + - Response: extracted markdown content for one document +- `PATCH /documents/{document_id}` + - Body model: `DocumentUpdateRequest` + - Response model: `DocumentResponse` +- `POST /documents/{document_id}/trash` + - Response model: `DocumentResponse` +- `POST /documents/{document_id}/restore` + - Response model: `DocumentResponse` +- `DELETE /documents/{document_id}` + - Behavior: permanent delete, requires document to be trashed first + - Response: deletion counters +- `POST /documents/{document_id}/reprocess` + - Response model: `DocumentResponse` + - Behavior: requeues asynchronous processing task + +### Upload + +- `POST /documents/upload` +- Multipart form fields: + - `files[]` (required) + - `relative_paths[]` (optional) + - `logical_path` (optional, defaults to `Inbox`) + - `tags` (optional CSV) + - `conflict_mode` (`ask`, `replace`, `duplicate`) +- Response model: `UploadResponse` +- Behavior: + - `ask`: returns `conflicts` if duplicate checksum is detected + - `replace`: creates new document linked to replaced document id + - `duplicate`: creates additional document record + +## Search + +- `GET /search` +- Query: `query` (min length 2), `offset`, `limit`, `include_trashed`, `only_trashed`, `path_filter`, `tag_filter`, `type_filter`, `processed_from`, `processed_to` +- Response model: `SearchResponse` +- Behavior: PostgreSQL full-text and metadata ranking + +## Processing Logs + +- `GET /processing/logs` + - Query: `offset`, `limit`, `document_id` + - Response model: `ProcessingLogListResponse` +- `POST /processing/logs/trim` + - Query: `keep_document_sessions`, `keep_unbound_entries` + - Response: trim counters +- `POST /processing/logs/clear` + - Response: clear counters + +## Settings + +- `GET /settings` + - Response model: `AppSettingsResponse` +- `PATCH /settings` + - Body model: `AppSettingsUpdateRequest` + - Response model: `AppSettingsResponse` +- `POST /settings/reset` + - Response model: `AppSettingsResponse` +- `PATCH /settings/handwriting` + - Body model: `HandwritingSettingsUpdateRequest` + - Response model: `AppSettingsResponse` +- `GET /settings/handwriting` + - Response model: `HandwritingSettingsResponse` + +## Schema Families + +Document schemas in `backend/app/schemas/documents.py`: +- `DocumentResponse` +- `DocumentDetailResponse` +- `DocumentsListResponse` +- `UploadConflict` +- `UploadResponse` +- `DocumentUpdateRequest` +- `SearchResponse` +- `ContentExportRequest` + +Processing log schemas in `backend/app/schemas/processing_logs.py`: +- `ProcessingLogEntryResponse` +- `ProcessingLogListResponse` + +Settings schemas in `backend/app/schemas/settings.py`: +- Provider, task, upload-default, display, predefined paths or tags, handwriting-style, and legacy handwriting models grouped under `AppSettingsResponse` and `AppSettingsUpdateRequest`. diff --git a/doc/architecture-overview.md b/doc/architecture-overview.md new file mode 100644 index 0000000..99b51f4 --- /dev/null +++ b/doc/architecture-overview.md @@ -0,0 +1,66 @@ +# Architecture Overview + +## System Topology + +DMS runs as a multi-service application defined in `docker-compose.yml`: +- `frontend` serves the React UI on port `5173` +- `api` serves FastAPI on port `8000` +- `worker` executes asynchronous extraction and indexing jobs +- `db` provides PostgreSQL persistence on port `5432` +- `redis` backs queueing on port `6379` +- `typesense` stores search index and vector-adjacent metadata on port `8108` + +## Backend Architecture + +Backend source root: `backend/app/` + +Main boundaries: +- `api/` route handlers and HTTP contract +- `services/` domain logic (storage, extraction, routing, settings, processing logs, Typesense) +- `db/` SQLAlchemy base, engine, and session lifecycle +- `models/` persistence entities (`Document`, `ProcessingLogEntry`) +- `schemas/` Pydantic response and request schemas +- `worker/` RQ queue integration and background processing tasks + +Application bootstrap in `backend/app/main.py`: +- mounts routers under `/api/v1` +- configures CORS from settings +- initializes storage, settings, database schema, and Typesense collection on startup + +## Processing Lifecycle + +1. Upload starts at `POST /api/v1/documents/upload`. +2. API stores file bytes and inserts document rows with status `queued`. +3. API enqueues `app.worker.tasks.process_document_task` into Redis. +4. Worker extracts content and metadata, handles ZIP expansion, runs OCR and routing suggestions, and writes processing logs. +5. Worker updates database fields, document status, and search index entries. +6. UI polls for documents and processing logs to reflect progress. + +## Frontend Architecture + +Frontend source root: `frontend/src/` + +Core structure: +- `App.tsx` orchestrates screen switching, state, polling, and action flows +- `components/` contains upload, filter, grid, viewer, modal, settings, and log panel modules +- `lib/api.ts` centralizes API client calls +- `types.ts` defines typed API contracts used by components +- `design-foundation.css` and `styles.css` define design tokens and global/component styling + +Main user flows: +- Upload and conflict resolution +- Search and filtered document browsing +- Metadata editing and lifecycle actions (trash, restore, delete, reprocess) +- Settings management for providers, tasks, and UI defaults +- Processing log review + +## Persistence and State + +Persistent data: +- PostgreSQL stores document metadata and processing logs +- Docker volume-backed storage keeps original files, previews, and settings JSON +- Typesense stores indexed search representations + +Transient runtime state: +- Redis queues processing tasks and worker execution state +- frontend local component state drives active filters, selection, and modal flows diff --git a/doc/data-model-reference.md b/doc/data-model-reference.md new file mode 100644 index 0000000..9494125 --- /dev/null +++ b/doc/data-model-reference.md @@ -0,0 +1,53 @@ +# Data Model Reference + +Primary SQLAlchemy models are defined in `backend/app/models/`. + +## documents + +Model: `Document` in `backend/app/models/document.py` + +Purpose: +- Stores source file identity, storage location, extracted content, lifecycle status, and classification metadata. + +Core fields: +- Identity and source: `id`, `original_filename`, `source_relative_path`, `stored_relative_path` +- File attributes: `mime_type`, `extension`, `sha256`, `size_bytes` +- Organization: `logical_path`, `suggested_path`, `tags`, `suggested_tags` +- Processing outputs: `extracted_text`, `image_text_type`, `handwriting_style_id`, `preview_available` +- Lifecycle and relations: `status`, `is_archive_member`, `archived_member_path`, `parent_document_id`, `replaces_document_id` +- Metadata and timestamps: `metadata_json`, `created_at`, `processed_at`, `updated_at` + +Enum `DocumentStatus`: +- `queued` +- `processed` +- `unsupported` +- `error` +- `trashed` + +Relationships: +- Self-referential `parent_document` relationship for archive extraction trees. + +## processing_logs + +Model: `ProcessingLogEntry` in `backend/app/models/processing_log.py` + +Purpose: +- Stores timestamped pipeline events for upload, extraction, OCR, routing, indexing, and errors. + +Core fields: +- Event identity and timing: `id`, `created_at` +- Event classification: `level`, `stage`, `event` +- Document linkage: `document_id`, `document_filename` +- Model context: `provider_id`, `model_name` +- Prompt or response traces: `prompt_text`, `response_text` +- Structured event payload: `payload_json` + +Foreign keys: +- `document_id` references `documents.id` with `ON DELETE SET NULL`. + +## Model Lifecycle Notes + +- Upload inserts a `Document` row in `queued` state and enqueues background processing. +- Worker updates extraction results and final status (`processed`, `unsupported`, or `error`). +- Trash and restore operations toggle `status` while preserving source files until permanent delete. +- Permanent delete removes the document tree (including archive descendants) and associated stored files. diff --git a/doc/operations-and-configuration.md b/doc/operations-and-configuration.md new file mode 100644 index 0000000..5b1c4bb --- /dev/null +++ b/doc/operations-and-configuration.md @@ -0,0 +1,111 @@ +# Operations And Configuration + +## Runtime Services + +`docker-compose.yml` defines the runtime stack: +- `db` (Postgres 16, port `5432`) +- `redis` (Redis 7, port `6379`) +- `typesense` (Typesense 29, port `8108`) +- `api` (FastAPI backend, port `8000`) +- `worker` (RQ background worker) +- `frontend` (Vite UI, port `5173`) + +## Named Volumes + +Persistent volumes: +- `db-data` +- `redis-data` +- `dcm-storage` +- `typesense-data` + +Reset all persisted runtime data: + +```bash +docker compose down -v +``` + +## Operational Commands + +Start or rebuild stack: + +```bash +docker compose up --build -d +``` + +Stop stack: + +```bash +docker compose down +``` + +Tail logs: + +```bash +docker compose logs -f +``` + +## Backend Configuration + +Settings source: +- Runtime settings class: `backend/app/core/config.py` +- API settings persistence: `backend/app/services/app_settings.py` + +Key environment variables used by `api` and `worker` in compose: +- `APP_ENV` +- `DATABASE_URL` +- `REDIS_URL` +- `STORAGE_ROOT` +- `PUBLIC_BASE_URL` +- `CORS_ORIGINS` (API service) +- `TYPESENSE_PROTOCOL` +- `TYPESENSE_HOST` +- `TYPESENSE_PORT` +- `TYPESENSE_API_KEY` +- `TYPESENSE_COLLECTION_NAME` + +Selected defaults from `Settings` (`backend/app/core/config.py`): +- `upload_chunk_size = 4194304` +- `max_zip_members = 250` +- `max_zip_depth = 2` +- `max_text_length = 500000` +- `default_openai_model = "gpt-4.1-mini"` +- `default_openai_timeout_seconds = 45` +- `default_summary_model = "gpt-4.1-mini"` +- `default_routing_model = "gpt-4.1-mini"` +- `typesense_timeout_seconds = 120` +- `typesense_num_retries = 0` + +## Frontend Configuration + +Frontend runtime API target: +- `VITE_API_BASE` in `docker-compose.yml` frontend service + +Frontend local commands: + +```bash +cd frontend && npm run dev +cd frontend && npm run build +cd frontend && npm run preview +``` + +## Settings Persistence + +Application-level settings managed from the UI are persisted by backend settings service: +- file path: `/settings.json` +- endpoints: `/api/v1/settings`, `/api/v1/settings/reset`, `/api/v1/settings/handwriting` + +Settings include: +- upload defaults +- display options +- provider configuration +- OCR, summary, and routing task settings +- predefined paths and tags +- handwriting-style clustering settings + +## Validation Checklist + +After operational or configuration changes, verify: +- `GET /api/v1/health` is healthy +- frontend can list, upload, and search documents +- processing worker logs show successful task execution +- settings save or reset works and persists after restart