From 08efe511e5324af55025502b5efac55a7db54aab Mon Sep 17 00:00:00 2001 From: Alaashamel Date: Thu, 20 Aug 2026 03:27:56 +0300 Subject: [PATCH] docs: architecture diagrams and comprehensive API/AI/database documentation Add 8 Mermaid architecture diagrams: system overview, request lifecycle, authentication flow, RBAC permission matrix, AI pipeline flow, real-time Socket.io architecture, data model ERD, and deployment topology. Add comprehensive API reference (docs/api/README.md) covering all 85 HTTP endpoints and 6 Socket.io events across 15 modules: auth, organizations, projects, milestones, labels, tasks, GitHub OAuth, repositories, analytics, notifications, activity, chat, AI, and real-time. Includes request/response examples, error codes, pagination, and permission matrix. Add AI service documentation (docs/ai/README.md) covering pipeline architecture, provider gateway, security model (prompt injection defense, secret redaction), ingestion pipeline, RAG retrieval, job intent protocol, engineering assistant SSE flow, and testing guide. Add database documentation (docs/database/README.md) covering schema overview, 12 migrations, entity relationships, seed data, Redis keys, pgvector setup, and performance indexes. --- docs/ai/README.md | 241 ++++++++++ docs/api/README.md | 820 ++++++++++++++++++++++++++++++++++ docs/architecture/diagrams.md | 346 ++++++++++++++ docs/database/README.md | 229 ++++++++++ 4 files changed, 1636 insertions(+) create mode 100644 docs/ai/README.md create mode 100644 docs/api/README.md create mode 100644 docs/architecture/diagrams.md create mode 100644 docs/database/README.md diff --git a/docs/ai/README.md b/docs/ai/README.md new file mode 100644 index 0000000..465f0ce --- /dev/null +++ b/docs/ai/README.md @@ -0,0 +1,241 @@ +# AI Service Documentation + +`apps/ai` — Python/FastAPI service for repository ingestion, embeddings, +vector search (RAG) and model invocation. + +--- + +## Quick Start + +```bash +# Development +cd apps/ai +pip install -e ".[dev]" +uvicorn app.main:app --reload --port 5001 + +# Tests +python -m pytest # all tests +python -m pytest tests/test_analyzer.py # analyzer tests only +python -m ruff check . # lint +``` + +**Environment variables:** + +| Variable | Required | Description | +|----------|----------|-------------| +| `AI_PRIMARY_PROVIDER` | Yes | `openai`, `anthropic`, or `local` | +| `AI_OPENAI_API_KEY` | If OpenAI | OpenAI API key | +| `AI_ANTHROPIC_API_KEY` | If Anthropic | Anthropic API key | +| `AI_JOB_SECRET` | Yes | HMAC secret (must match API) | +| `AI_DATABASE_URL` | Yes | PostgreSQL connection string | + +--- + +## Capabilities + +| Capability | Analysis Type | Input | Output | +|------------|--------------|-------|--------| +| **Repository Analysis** | `analyzer` | Repo archive (tarball) | 4 health scores + strengths/risks/recommendations | +| **Code Review** | `code_review` | PR diff (inline) | Severity-classified findings + review score | +| **Documentation Generator** | `docs`, `readme` | Repo archive | Markdown draft files | +| **Engineering Assistant** | — | User question + repo context | Streamed text reply with sources | + +--- + +## Pipeline Architecture + +Every capability follows the same pipeline pattern: + +``` +Input contract + → Context assembly (what does the model need to see?) + → Prompt construction (instruction layer, versioned) + → Provider invocation (model call via gateway) + → Structured output (JSON schema enforced) + → Validation (Pydantic; reject malformed output) + → Application logic (scoring, aggregation, mapping) + → Persistence/return (typed job result) +``` + +**Design principles:** +- **Deterministic where possible:** filtering, language detection, and + scoring are code, not model output. +- **Model output is data, not code:** never executed; always + schema-checked. +- **Previews, not overwrites:** generation pipelines return drafts that + the user explicitly approves. + +--- + +## Provider Gateway + +```python +# apps/ai/app/providers/base.py +class ModelGateway: + def complete(self, request: CompletionRequest) -> CompletionResult: ... + +class OpenAIAdapter(ModelGateway): ... +class AnthropicAdapter(ModelGateway): ... +class LocalAdapter(ModelGateway): ... +``` + +- Provider is selected via `AI_PRIMARY_PROVIDER` env var. +- A fallback provider is used for resilience. +- Providers are exercised only against bounded payloads; repository + content is chunked and summarized before it reaches a model. +- No API keys are exposed beyond the AI service environment. + +### Supported Providers + +| Provider | Embedding Model | Completion Model | Notes | +|----------|----------------|------------------|-------| +| OpenAI | `text-embedding-3-small` | `gpt-4o-mini` | Default provider | +| Anthropic | — | `claude-3-haiku` | No native embeddings; uses hashing fallback | +| Local | `hashing_embed` | — | Deterministic offline embedder for testing | + +--- + +## Security Model + +- **Prompt injection defense:** external content (repo files, PR diffs) + is wrapped in `…` tags and insulated from + instructions. +- **Secret redaction:** repository content is scanned for obvious secret + patterns before ingestion. Redacted chunks are never sent to a model. +- **No secrets to models:** GitHub tokens never reach the AI service. + Archives are pulled from a signed API endpoint. +- **Job tokens:** all inbound requests authenticated via API-signed + HMAC-SHA256 tokens (`X-Devforge-Job-Token` header). +- **Never execute AI output:** all model responses are validated as data, + never evaluated as code. + +--- + +## Repository Ingestion Pipeline + +``` +1. Fetch tarball from signed archive URL +2. Decode paths (percent-decoded) +3. Filter: ignore node_modules, build output, binaries, VCS +4. Language detection (per-extension mapping) +5. Dependency manifest parsing (package.json, requirements.txt, etc.) +6. Chunk files into retrieval units with metadata +7. Secret scan + redact sensitive patterns +8. Embed chunks via provider (pgvector storage) +9. Store in ai_document_chunks (1536-dim HNSW index) +``` + +--- + +## RAG Retrieval + +Hybrid search combining: + +1. **Vector search:** cosine similarity against pgvector embeddings +2. **Keyword search:** PostgreSQL full-text search on chunk content + +Results are merged, deduplicated, and assembled into a token-budgeted +context window for the model. + +**Chunk metadata stored:** path, language, line range, repository ID. + +--- + +## Job Intent Protocol + +The API submits bounded job intents — never raw repository content: + +```json +{ + "job_id": "uuid", + "type": "analyzer | code_review | docs | readme", + "organization_id": "uuid", + "repository_id": "uuid", + "archive_url": "http://api:4000/api/v1/ai/archive/:repoId?token=...", + "archive_token": "...", + "payload": { + "repository_name": "acme/web", + "pull_request_number": 42 + } +} +``` + +**Flow:** +1. API creates `ai_jobs` row (`queued`), signs HMAC job token +2. API POSTs intent to `AI_SERVICE_URL/jobs/{jobId}` +3. AI service verifies token, pulls archive (if needed), processes +4. AI service persists results to `ai_analyses` and updates `ai_jobs` +5. API polls job status and returns results to the browser + +**Code review variant:** skips archive flow; API fetches PR diff from +GitHub and submits it inline in `payload.diff`. + +**Assistant variant:** stateless SSE endpoint (`POST /assistant/stream`); +not a queued job. + +--- + +## Engineering Assistant + +The assistant is a stateless, streamed endpoint: + +``` +POST {AI_SERVICE_URL}/assistant/stream +``` + +Body: + +```json +{ + "conversation_id": "uuid", + "organization_id": "uuid", + "repository_id": "uuid", + "repository_name": "acme/web", + "messages": [ + { "role": "user", "content": "How is auth structured?" } + ] +} +``` + +**Response: Server-Sent Events** + +``` +event: sources +data: [{"path":"src/auth/routes.js","chunk":"..."}] + +event: delta +data: "The auth module uses JWT tokens..." + +event: delta +data: " with Argon2id password hashing..." + +event: done +data: {} +``` + +- Conversation history is truncated to the last 20 messages. +- Repository chunks are scoped to a single repository. +- Excerpts are wrapped in `` tags as data-only context. +- The API persists the assistant reply (with sources) only on successful + stream completion. + +--- + +## Testing + +```bash +python -m pytest # all 145+ tests +python -m pytest -v # verbose +python -m pytest tests/test_auth.py # auth/token tests +python -m pytest -k "analyzer" # analyzer tests +``` + +**Test categories:** +- **Unit:** pipelines with mocked providers (pytest) +- **Contract:** API ↔ AI job protocol (`test_auth.py`) +- **Integration:** real provider calls behind a feature flag + +--- + +*See also: [AI service architecture](../architecture/ai-service-architecture.md) · +[ADR-003](../decisions/ADR-003-ai-service-boundary.md)* diff --git a/docs/api/README.md b/docs/api/README.md new file mode 100644 index 0000000..d1d069c --- /dev/null +++ b/docs/api/README.md @@ -0,0 +1,820 @@ +# DevForge API Reference + +Base URL: `http://localhost:4000` (development) or `https://api.example.com` (production). + +All responses use a consistent envelope: + +```json +{ "data": { ... } } +``` + +Errors: + +```json +{ "error": { "code": "VALIDATION_ERROR", "message": "Invalid input", "details": [...] } } +``` + +--- + +## Table of Contents + +- [Authentication](#authentication) +- [Pagination & Filtering](#pagination--filtering) +- [Error Codes](#error-codes) +- [Health & Metrics](#health--metrics) +- [Auth Module](#auth-module) +- [Organizations](#organizations) +- [Projects](#projects) +- [Milestones](#milestones) +- [Labels](#labels) +- [Tasks](#tasks) +- [GitHub OAuth](#github-oauth) +- [Repositories](#repositories) +- [Analytics](#analytics) +- [Notifications](#notifications) +- [Activity Feed](#activity-feed) +- [Chat](#chat) +- [AI Module](#ai-module) +- [Real-Time (Socket.io)](#real-time-socketio) + +--- + +## Authentication + +All authenticated requests require a `Authorization: Bearer ` +header. Access tokens are short-lived HS256 JWTs (default 15 minutes). + +Obtain tokens via `POST /api/v1/auth/login` or `POST /api/v1/auth/register`. +Refresh expired tokens via `POST /api/v1/auth/refresh`. + +```bash +curl -H "Authorization: Bearer eyJ..." http://localhost:4000/api/v1/organizations +``` + +--- + +## Pagination & Filtering + +List endpoints support: + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `page` | 1 | Page number | +| `limit` | 20 | Items per page | +| `sort` | `created_at` | Sort field (direction via `-` prefix) | +| `q` | — | Full-text search | + +Response meta: + +```json +{ + "data": [...], + "meta": { "total": 42, "page": 1, "limit": 20 } +} +``` + +--- + +## Error Codes + +| Code | HTTP Status | Description | +|------|-------------|-------------| +| `VALIDATION_ERROR` | 400 | Request body or params failed validation | +| `UNAUTHORIZED` | 401 | Missing or invalid access token | +| `FORBIDDEN` | 403 | Authenticated but lacks required permission | +| `NOT_FOUND` | 404 | Resource not found | +| `CONFLICT` | 409 | Duplicate resource (e.g. unique constraint) | +| `RATE_LIMITED` | 429 | Too many requests (retry after `Retry-After` header) | +| `INTERNAL_ERROR` | 500 | Unexpected server error | + +--- + +## Health & Metrics + +### Liveness + +``` +GET /api/v1/health +GET /api/v1/health/live +``` + +Returns 200 with `{ status: "ok" }`. + +### Readiness + +``` +GET /api/v1/health/ready +``` + +Returns 200 when the database is reachable, 503 `"degraded"` otherwise. + +### Prometheus Metrics + +``` +GET /metrics +``` + +Returns Prometheus text format. No authentication required. + +--- + +## Auth Module + +`/api/v1/auth` + +All auth endpoints are rate-limited per IP. + +### Register + +``` +POST /api/v1/auth/register +``` + +```json +{ + "email": "user@example.com", + "password": "SecureP@ss123", + "name": "Jane Doe" +} +``` + +**Response 201:** + +```json +{ + "data": { + "user": { "id": "...", "email": "user@example.com", "name": "Jane Doe" }, + "verificationToken": "abc123..." + } +} +``` + +> In production, the verification token is sent via email and never +> returned in the response. + +### Login + +``` +POST /api/v1/auth/login +``` + +```json +{ "email": "user@example.com", "password": "SecureP@ss123" } +``` + +**Response 200:** + +```json +{ + "data": { + "accessToken": "eyJ...", + "refreshToken": "opaque-rotating-token", + "user": { "id": "...", "email": "user@example.com" } + } +} +``` + +### Refresh Token + +``` +POST /api/v1/auth/refresh +``` + +```json +{ "refreshToken": "opaque-rotating-token" } +``` + +Returns a new `accessToken` + `refreshToken` pair. The old refresh token +is revoked (rotation with reuse detection). + +### Logout + +``` +POST /api/v1/auth/logout +``` + +```json +{ "refreshToken": "..." } +``` + +### Verify Email + +``` +POST /api/v1/auth/verify-email +``` + +```json +{ "token": "abc123..." } +``` + +### Forgot Password + +``` +POST /api/v1/auth/forgot-password +``` + +```json +{ "email": "user@example.com" } +``` + +### Reset Password + +``` +POST /api/v1/auth/reset-password +``` + +```json +{ "token": "abc123...", "password": "NewP@ss456" } +``` + +### Get Current User + +``` +GET /api/v1/auth/me +``` + +**Response 200:** + +```json +{ + "data": { + "id": "...", + "email": "user@example.com", + "name": "Jane Doe", + "avatarUrl": "...", + "emailVerifiedAt": "2026-01-15T10:00:00Z", + "status": "active" + } +} +``` + +--- + +## Organizations + +### List My Organizations + +``` +GET /api/v1/organizations +``` + +**Response 200:** + +```json +{ + "data": [ + { + "id": "...", + "name": "Acme Labs", + "slug": "acme-labs", + "role": "admin", + "memberCount": 12, + "projectCount": 5 + } + ] +} +``` + +### List Organization Members + +``` +GET /api/v1/organizations/:orgId/members +``` + +Requires: `project.view` + +--- + +## Projects + +Base: `/api/v1/organizations/:orgId/projects` + +### List Projects + +``` +GET …/projects +``` + +Requires: `project.view` + +### Create Project + +``` +POST …/projects +``` + +```json +{ + "name": "DevForge", + "key": "DF", + "description": "AI-powered developer platform", + "defaultPriority": "medium" +} +``` + +Requires: `projects.create` + +### Get Project + +``` +GET …/projects/:projectId +``` + +### Update Project + +``` +PATCH …/projects/:projectId +``` + +Requires: `projects.manage` + +### Delete Project + +``` +DELETE …/projects/:projectId +``` + +Requires: `projects.delete` + +### Project Members + +``` +GET …/projects/:projectId/members +PUT …/projects/:projectId/members/:userId { "role": "developer" } +DELETE …/projects/:projectId/members/:userId +``` + +Requires: `projects.manage` (write), `project.view` (read) + +--- + +## Milestones + +Base: `…/projects/:projectId/milestones` + +``` +GET …/milestones +POST …/milestones { "title", "description", "dueDate", "status" } +PATCH …/milestones/:milestoneId +DELETE …/milestones/:milestoneId +``` + +Requires: `project.view` (read), `projects.manage` (write) + +Milestone statuses: `planned`, `active`, `completed`, `cancelled` + +--- + +## Labels + +Base: `…/projects/:projectId/labels` + +``` +GET …/labels +POST …/labels { "name", "color": "#ff0000" } +PATCH …/labels/:labelId +DELETE …/labels/:labelId +``` + +Requires: `project.view` (read), `projects.manage` (write) + +Colors must be valid hex (`#rrggbb`). Duplicate names per project return +409. + +--- + +## Tasks + +Base: `…/projects/:projectId/tasks` + +### CRUD + +``` +GET …/tasks ?status=open&priority=high&q=search&sort=-created_at&page=1&limit=20 +POST …/tasks { "title", "type", "status", "priority", "assigneeId", "milestoneId", "dueDate", "estimate" } +GET …/tasks/:taskId +PATCH …/tasks/:taskId { "status": "in_progress", "priority": "urgent" } +DELETE …/tasks/:taskId +``` + +Task types: `task`, `issue`, `bug` + +Filters: `?status=`, `?priority=`, `?type=`, `?assigneeId=`, `?milestoneId=`, `?label=` + +Requires: `project.view` (read), `tasks.manage` (write) + +### Comments + +``` +GET …/tasks/:taskId/comments +POST …/tasks/:taskId/comments { "body": "Looks good!" } +PATCH …/tasks/:taskId/comments/:commentId { "body": "Updated" } +DELETE …/tasks/:taskId/comments/:commentId +``` + +### Labels + +``` +PUT …/tasks/:taskId/labels { "labels": [{ "id": "...", "name": "bug", "color": "#ff0000" }] } +``` + +### Activity + +``` +GET …/tasks/:taskId/activity +``` + +Returns append-only audit events: `created`, `status_change`, +`priority_change`, `labels_change`, `comment`, `dependency_added`, etc. + +### Dependencies + +``` +GET …/tasks/:taskId/dependencies +POST …/tasks/:taskId/dependencies { "dependsOnId": "..." } +DELETE …/tasks/:taskId/dependencies/:dependsOnId +``` + +Self-dependencies, cross-project dependencies, and cycles are rejected +with 409. + +--- + +## GitHub OAuth + +### Begin OAuth Flow + +``` +POST /api/v1/github/oauth/begin +``` + +**Response 200:** + +```json +{ "data": { "url": "https://github.com/login/oauth/authorize?client_id=..." } } +``` + +### OAuth Callback + +``` +GET /api/v1/github/oauth/callback?code=...&state=... +``` + +Handled automatically by GitHub redirect. Returns 302 to the web app +with `?github=connected` or `?github=error`. + +### Connection Status + +``` +GET /api/v1/github/connection +``` + +### Disconnect + +``` +POST /api/v1/github/disconnect +``` + +--- + +## Repositories + +Base: `/api/v1/organizations/:orgId/repositories` + +### Import + +``` +POST …/repositories/import { "owner": "acme", "name": "web" } +``` + +Requires: `repos.manage` + +### List / Get + +``` +GET …/repositories +GET …/repositories/:repoId +``` + +Requires: `project.view` + +### Sync + +``` +POST …/repositories/:repoId/sync +``` + +Refreshes metadata and pull requests from GitHub. Requires: `repos.manage` + +### Delete + +``` +DELETE …/repositories/:repoId +``` + +### Pull Requests + +``` +GET …/repositories/:repoId/pull-requests ?state=open&page=1&limit=20 +``` + +### Branches + +``` +GET …/repositories/:repoId/branches +``` + +### Commits + +``` +GET …/repositories/:repoId/commits ?sha=main&limit=30 +``` + +### Issues + +``` +GET …/repositories/:repoId/issues ?state=open +``` + +### Webhooks + +``` +GET …/repositories/:repoId/webhooks +POST …/repositories/:repoId/webhooks { "events": ["push", "pull_request"] } +DELETE …/repositories/:repoId/webhooks/:webhookId +``` + +Requires: `project.view` (read), `repos.manage` (write) + +--- + +## Analytics + +Base: `/api/v1/organizations/:orgId/analytics` + +All endpoints require: `project.view` + +### Overview + +``` +GET …/analytics/overview +``` + +Returns counts (projects, tasks, repos, members), additions/deletions, +completion ratio, top contributors, recently merged PRs. + +### Velocity + +``` +GET …/analytics/velocity ?weeks=12 +``` + +Weekly buckets (Monday-aligned) of merged PRs, done tasks, closed +issues, and completed reviews. + +### Health + +``` +GET …/analytics/health +``` + +Weighted 0–100 score with `healthy` / `degraded` / `critical` / +`no_data` status. Components: task completion, merge rate, issue close +rate, review coverage. + +### Developers + +``` +GET …/analytics/developers ?weeks=12 +``` + +Per-member task + PR aggregation with velocity points and health scores. + +### Repositories + +``` +GET …/analytics/repositories +``` + +Summary stats for all imported repositories. + +### Repository Activity + +``` +GET …/analytics/repositories/:repoId/activity ?months=6 +``` + +Monthly, recent, and review-by-status breakdowns. + +--- + +## Notifications + +`/api/v1/notifications` + +``` +GET /api/v1/notifications ?limit=50&unread=true +GET /api/v1/notifications/unread-count +POST /api/v1/notifications/read-all +POST /api/v1/notifications/:id/read +``` + +Notifications are user-scoped (derived from the access token). Types +include `task_assigned`, `task_updated`, `task_commented`, +`notification:new` (real-time push). + +--- + +## Activity Feed + +`/api/v1/organizations/:orgId/activity` + +``` +GET …/activity ?limit=50 +``` + +Append-only feed of task events across the organization. Requires: +`project.view` + +--- + +## Chat + +`/api/v1/organizations/:orgId/chat` + +``` +GET …/chat/messages ?before=&limit=50 +POST …/chat/messages { "body": "Hello team!" } +``` + +Requires: `project.view` + +Messages are persisted in `chat_messages` and broadcast in real-time +via Socket.io to the `chat:{orgId}` room. + +--- + +## AI Module + +Base: `/api/v1/organizations/:orgId/ai` + +### Create Analysis + +``` +POST …/ai/analyses +``` + +```json +{ + "type": "analyzer", + "repositoryId": "..." +} +``` + +Analysis types: `analyzer`, `code_review`, `docs`, `readme` + +For `code_review`, include `pullRequestNumber`: + +```json +{ + "type": "code_review", + "repositoryId": "...", + "pullRequestNumber": 42 +} +``` + +Requires: `ai.run` + +**Response 201:** + +```json +{ + "data": { + "jobId": "...", + "status": "queued" + } +} +``` + +### List Analyses + +``` +GET …/ai/analyses ?type=analyzer&pullRequestNumber=42 +``` + +### Get Analysis + +``` +GET …/ai/analyses/:analysisId +``` + +### Approve Analysis Result (Docs/README) + +``` +POST …/ai/analyses/:analysisId/approve +``` + +```json +{ + "filePath": "README.md", + "repositoryId": "..." +} +``` + +Commits the generated file to GitHub via the Contents API. The approval +is recorded in `report.approvals`. + +### Job Status + +``` +GET …/ai/jobs/:jobId +``` + +Poll until `status` is `succeeded` or `failed`. + +### Conversations + +``` +GET …/ai/conversations +POST …/ai/conversations { "title": "Architecture Q&A", "repositoryId": "..." } +GET …/ai/conversations/:conversationId +DELETE …/ai/conversations/:conversationId +``` + +### Messages + +``` +GET …/ai/conversations/:conversationId/messages +``` + +### Stream Assistant Reply + +``` +POST …/ai/conversations/:conversationId/stream +``` + +```json +{ + "repositoryId": "...", + "message": "How is the auth module structured?" +} +``` + +Returns Server-Sent Events: + +``` +event: sources +data: [{"path":"src/modules/auth/routes.js","chunk":"..."}] + +event: delta +data: "The auth module uses..." + +event: delta +data: " JWT tokens with..." + +event: done +data: {} +``` + +Requires: `ai.run` + +--- + +## Real-Time (Socket.io) + +Namespace: `/realtime` + +### Connection + +```javascript +const socket = io('/realtime', { auth: { accessToken: 'eyJ...' } }); +``` + +### Client → Server Events + +| Event | Payload | Description | +|-------|---------|-------------| +| `room:join` | `{ room: 'org:{id}' }` | Join a room (server-verified) | +| `room:leave` | `{ room: 'org:{id}' }` | Leave a room | +| `presence:join` | `{ orgId }` | Set online presence | +| `presence:heartbeat` | `{ orgId }` | Keep presence alive (90s TTL) | +| `chat:typing` | `{ orgId }` | Emit typing indicator (2s throttle) | + +### Server → Client Events + +| Event | Room | Payload | +|-------|------|---------| +| `notification:new` | `user:{id}` | `{ id, type, title, body, href }` | +| `activity:new` | `org:{id}` | `{ id, type, actor, subject }` | +| `presence:update` | `org:{id}` | `{ onlineUserIds: [...] }` | +| `task:created` | `project:{id}` | `{ task }` | +| `task:updated` | `project:{id}` / `task:{id}` | `{ task, changes }` | +| `task:comment` | `task:{id}` | `{ comment }` | +| `chat:message` | `chat:{orgId}` | `{ message }` | +| `chat:typing` | `chat:{orgId}` | `{ userId, orgId }` | + +--- + +*See also: [architecture diagrams](../architecture/diagrams.md) · +[api design](../architecture/api-design.md) · +[backend architecture](../architecture/backend-architecture.md)* diff --git a/docs/architecture/diagrams.md b/docs/architecture/diagrams.md new file mode 100644 index 0000000..87a38d4 --- /dev/null +++ b/docs/architecture/diagrams.md @@ -0,0 +1,346 @@ +# Architecture Diagrams + +Visual diagrams for the DevForge system. All diagrams use +[Mermaid](https://mermaid.js.org/) syntax and render in GitHub's +markdown viewer. + +--- + +## 1. System Architecture + +High-level view of how the five services connect. + +```mermaid +graph TB + Browser["Browser (React SPA)"] + + subgraph "Docker Compose Stack" + Nginx["nginx :8080"] + Web["web :80 (internal)"] + API["api :4000"] + AI["ai :5001"] + PG[("postgres :5432\npgvector")] + end + + Browser -->|"HTTP / HTTPS"| Nginx + Nginx -->|"/ → static files"| Web + Nginx -->|"/api/v1/*"| API + Nginx -->|"/socket.io/*"| API + Nginx -->|"/metrics"| API + + API -->|"SQL + pgvector"| PG + AI -->|"SQL + pgvector"| PG + + API -->|"POST /jobs/{id}\n(signed job intent)"| AI + AI -->|"GET /api/v1/ai/archive/:repoId\n(signed archive pull)"| API + + subgraph "External" + GitHub["GitHub API\nOAuth + Webhooks"] + LLM["AI Providers\nOpenAI / Anthropic / Local"] + end + + API -->|"OAuth + REST\n(encrypted tokens)"| GitHub + GitHub -->|"Webhook push/PR"| API + AI -->|"Embeddings + Completion"| LLM +``` + +**Key invariant:** The browser never talks to the AI service directly. +All AI requests flow through the API, which signs job intents and holds +credentials server-side. + +--- + +## 2. Request Lifecycle + +How a typical API request is processed. + +```mermaid +sequenceDiagram + participant C as Browser + participant N as nginx + participant A as API + participant DB as PostgreSQL + participant R as Redis + + C->>N: GET /api/v1/orgs/:id/projects + N->>A: Proxy (path rewrite) + A->>A: requestId middleware + A->>A: requestLogger middleware + A->>A: metrics middleware (histogram) + A->>A: CORS check + A->>A: JWT verify (jose) + A->>DB: SELECT projects WHERE org_id = ? + DB-->>A: rows + A->>A: RBAC authorize(project.view) + A-->>C: 200 { data: [...], meta: { total, page } } +``` + +--- + +## 3. Authentication & Authorization Flow + +```mermaid +sequenceDiagram + participant U as User + participant API as API + participant DB as PostgreSQL + + Note over U,DB: Registration + U->>API: POST /auth/register {email, password, name} + API->>API: Argon2id hash password + API->>DB: INSERT user + verification_token + API-->>U: 201 { user, verificationToken (dev only) } + + Note over U,DB: Login + U->>API: POST /auth/login {email, password} + API->>DB: SELECT user by email + API->>API: Argon2id verify + API->>DB: INSERT refresh_token (SHA-256 hash) + API-->>U: 200 { accessToken, refreshToken } + + Note over U,DB: Authenticated request + U->>API: GET /organizations (Bearer ) + API->>API: jose.jwtVerify (HS256, 15m TTL) + API->>API: requireAuth + authorize(permission) + API-->>U: 200 { data: [...] } + + Note over U,DB: Token refresh + U->>API: POST /auth/refresh {refreshToken} + API->>DB: SELECT token_hash (SHA-256 lookup) + API->>API: Rotation: revoke old, issue new pair + API->>DB: UPDATE revoked_at + INSERT new token + API-->>U: 200 { accessToken, refreshToken } +``` + +**Refresh token rotation:** Replaying a rotated or revoked token +revokes the entire token family (reuse detection). + +--- + +## 4. RBAC Permission Matrix + +```mermaid +graph LR + subgraph "Roles (most → least permissive)" + Owner["Owner"] + Admin["Admin"] + Maintainer["Maintainer"] + Developer["Developer"] + Viewer["Viewer"] + end + + subgraph "Permissions" + P1["org.manage"] + P2["members.manage"] + P3["projects.create"] + P4["projects.manage"] + P5["projects.delete"] + P6["repos.manage"] + P7["tasks.manage"] + P8["ai.run"] + P9["project.view"] + end + + Owner --> P1 & P2 & P3 & P4 & P5 & P6 & P7 & P8 & P9 + Admin --> P2 & P3 & P4 & P5 & P6 & P7 & P8 & P9 + Maintainer --> P3 & P4 & P5 & P6 & P7 & P8 & P9 + Developer --> P7 & P8 & P9 + Viewer --> P9 +``` + +RBAC is composed: org role + project role → the more permissive wins. + +--- + +## 5. AI Pipeline Flow + +```mermaid +graph TD + A["User clicks 'Run Analysis'"] --> B["API creates ai_jobs row\nstatus: queued"] + B --> C["API signs job intent\nHMAC-SHA256 token"] + C --> D["POST /jobs/{jobId}\nto AI service"] + D --> E["AI verifies token"] + E --> F{"Job type?"} + + F -->|"analyzer"| G1["Ingest repo archive\n(chunks, redact secrets)"] + F -->|"code_review"| G2["Receive PR diff inline\n(no archive needed)"] + F -->|"docs / readme"| G3["Ingest repo archive"] + F -->|"assistant"| G4["Hybrid retrieval\n(vector + keyword)"] + + G1 --> H1["Build prompt\n(instruction layer v1)"] + G2 --> H2["Build prompt\n(classify findings)"] + G3 --> H3["Build prompt\n(generate markdown)"] + G4 --> H4["Build prompt\n(context from chunks)"] + + H1 --> I["Provider gateway\nOpenAI / Anthropic / Local"] + H2 --> I + H3 --> I + H4 --> I + + I --> J["Model response\n(JSON)"] + J --> K["Pydantic validation\n(structured output check)"] + K --> L["Application logic\n(scoring, aggregation)"] + L --> M["Persist to ai_analyses\nor stream SSE"] + + M --> N["API reads result\nvia ai_jobs polling"] + N --> O["Web renders analysis\nscores, findings, drafts"] +``` + +--- + +## 6. Real-Time Architecture (Socket.io) + +```mermaid +graph TB + Browser["Browser"] + + subgraph "API Server" + Auth["JWT Handshake\n(verify accessToken)"] + Hub["Socket.io Hub\n(single namespace)"] + Rooms["Room Manager\n(server-authorized)"] + Presence["Presence Tracker\n(90s TTL sweep)"] + end + + subgraph "Rooms" + R1["user:{userId}"] + R2["org:{orgId}"] + R3["project:{projectId}"] + R4["task:{taskId}"] + R5["chat:{orgId}"] + end + + Browser -->|"connect\n{ accessToken }"| Auth + Auth --> Hub + Hub --> Rooms + Hub --> Presence + + Hub -->|"notification:new"| R1 + Hub -->|"activity:new\npresence:update"| R2 + Hub -->|"task:created\ntask:updated\ntask:comment"| R3 & R4 + Hub -->|"chat:message\nchat:typing"| R5 + + subgraph "Events (Client → Server)" + E1["room:join\nroom:leave"] + E2["presence:join\npresence:heartbeat"] + E3["chat:typing"] + end + + E1 --> Rooms + E2 --> Presence + E3 --> Hub +``` + +**Rooms are server-authorized:** the hub verifies org/project membership +in the database before allowing a join. Client-side events are never +trusted for authorization. + +--- + +## 7. Data Model (ERD) + +Simplified entity-relationship diagram. See +[data-model.md](./data-model.md) for full column definitions. + +```mermaid +erDiagram + users ||--o{ refresh_tokens : "has" + users ||--o{ verification_tokens : "has" + users ||--o{ password_reset_tokens : "has" + users ||--o{ github_connections : "links" + users ||--o{ organization_members : "belongs to" + users ||--o{ project_members : "assigned to" + users ||--o{ developer_metrics : "tracked in" + + organizations ||--o{ organization_members : "has" + organizations ||--o{ teams : "has" + organizations ||--o{ projects : "contains" + organizations ||--o{ repositories : "owns" + organizations ||--o{ activities : "emits" + organizations ||--o{ ai_analyses : "analyzed" + + teams ||--o{ team_members : "has" + team_members }o--|| users : "member" + + projects ||--o{ project_members : "has" + projects ||--o{ milestones : "has" + projects ||--o{ tasks : "has" + projects ||--o{ labels : "has" + + milestones ||--o{ tasks : "groups" + + tasks ||--o{ task_labels : "tagged with" + tasks ||--o{ task_comments : "discussed in" + tasks ||--o{ task_activity : "audited" + tasks ||--o{ task_dependencies : "depends on" + tasks }o--|| users : "assigned to" + tasks }o--|| users : "reported by" + + labels ||--o{ task_labels : "applied to" + + repositories ||--o{ repository_webhooks : "listens to" + repositories ||--o{ pull_requests : "has" + repositories ||--o{ code_reviews : "reviewed" + repositories ||--o{ ai_analyses : "analyzed" + + pull_requests ||--o{ code_reviews : "reviewed in" + + users ||--o{ notifications : "receives" + users ||--o{ ai_conversations : "initiates" + + ai_conversations ||--o{ ai_messages : "contains" +``` + +--- + +## 8. Deployment Topology + +```mermaid +graph TB + subgraph "Production" + LB["Load Balancer\n(TLS termination)"] + + subgraph "Docker Compose" + Nginx["nginx\n:8080"] + Web["web\nReact SPA + nginx"] + API1["api instance 1\n:4000"] + API2["api instance 2\n:4000\n(scaled)"] + AI1["ai instance 1\n:5001"] + AI2["ai instance 2\n:5001\n(scaled)"] + PG[("PostgreSQL\npgvector")] + Redis[("Redis\nadapter")] + end + + LB --> Nginx + Nginx --> Web + Nginx --> API1 + Nginx --> API2 + API1 --> PG + API2 --> PG + AI1 --> PG + AI2 --> PG + API1 --> Redis + API2 --> Redis + end + + subgraph "External" + GitHub["GitHub\nOAuth + Webhooks"] + OpenAI["OpenAI / Anthropic"] + Monitoring["Prometheus\n+ Grafana"] + end + + API1 --> GitHub + AI1 --> OpenAI + API1 -->|"GET /metrics"| Monitoring +``` + +**Scaling notes:** +- API and AI services are stateless (JWT-based sessions). +- For multi-instance, swap in the Socket.io Redis adapter by setting + `REDIS_URL`. +- PostgreSQL can be replaced with a managed service (RDS, Supabase). + +--- + +*See also: [system-overview.md](./system-overview.md) · +[backend-architecture.md](./backend-architecture.md) · +[frontend-architecture.md](./frontend-architecture.md)* diff --git a/docs/database/README.md b/docs/database/README.md new file mode 100644 index 0000000..5340f3a --- /dev/null +++ b/docs/database/README.md @@ -0,0 +1,229 @@ +# Database Documentation + +PostgreSQL (with pgvector extension) is the system of record for all +persistent data. + +--- + +## Quick Reference + +```bash +npm run db:migrate # Apply pending migrations +npm run db:down # Roll back the last migration +npm run db:status # Show applied/pending migrations +npm run db:migrate:create # Create a new migration file +npm run db:seed # Seed demo data for local development +``` + +**Connection:** set `DATABASE_URL` in `.env` (default: +`postgres://devforge:devforge@localhost:5433/devforge`) + +Dev Postgres binds host port `5433` to avoid conflicts with local +Postgres on `5432`. + +--- + +## Schema Overview + +29 tables across 7 domains, implemented via 12 versioned migrations. + +### Domain Map + +| Domain | Tables | Migration | +|--------|--------|-----------| +| **Identity** | `users`, `refresh_tokens`, `verification_tokens`, `password_reset_tokens` | 0001–0002 | +| **Organizations** | `organizations`, `organization_members`, `teams`, `team_members` | 0003 | +| **Projects** | `projects`, `project_members`, `milestones`, `tasks`, `labels`, `task_labels`, `task_comments`, `task_activity`, `task_dependencies` | 0004–0006 | +| **GitHub** | `github_connections`, `repositories`, `repository_webhooks`, `pull_requests`, `code_reviews` | 0007–0008 | +| **Collaboration** | `notifications`, `activities`, `chat_messages` | 0007, 0009 | +| **Analytics** | `developer_metrics` | 0007 | +| **AI** | `ai_analyses`, `ai_conversations`, `ai_messages`, `ai_jobs`, `ai_document_chunks` | 0010–0012 | + +### Key Conventions + +- **Primary keys:** `uuid` (default `gen_random_uuid()`) +- **Timestamps:** `created_at`, `updated_at` on every table; + `deleted_at` for soft-deletable entities +- **Foreign keys:** `ON DELETE` chosen per relationship (restrict / + cascade / set null) +- **Text columns:** `text` with `CHECK` length constraints; enums as + constrained text +- **Soft delete** for user-facing content (orgs, projects, tasks) +- **Indexes** on every FK and hot query paths + +--- + +## Entity Relationship Diagram + +See [data model](../architecture/data-model.md) for full column +definitions. See [architecture diagrams](../architecture/diagrams.md) +for a Mermaid ERD. + +### Identity Chain + +``` +users + ├─< refresh_tokens (SHA-256 hashed, rotation chain) + ├─< verification_tokens (single-use, 24h TTL) + ├─< password_reset_tokens (single-use, 1h TTL) + └─< github_connections (AES-256-GCM encrypted tokens) +``` + +### Organization → Project Chain + +``` +organizations + ├─< organization_members (role-based, 5 roles) + ├─< teams >─ team_members + └─< projects + ├─< project_members + ├─< milestones (planned/active/completed/cancelled) + ├─< tasks + │ ├─< task_labels + │ ├─< task_comments + │ ├─< task_activity (append-only audit) + │ └─< task_dependencies (cycle-checked) + └─< labels +``` + +### GitHub Chain + +``` +repositories + ├─< repository_webhooks (HMAC-signed secrets) + ├─< pull_requests (synced from GitHub) + └─< code_reviews (AI-generated) +``` + +### AI Chain + +``` +ai_analyses (per-repository, typed reports) +ai_conversations >─< ai_messages (assistant Q&A) +ai_jobs (async job tracking) +ai_document_chunks (pgvector embeddings, HNSW indexed) +``` + +--- + +## Migrations + +All migrations live in `packages/database/migrations/` and export +`up(db)` / `down(db)` functions applied inside a single transaction. + +| # | Name | Purpose | +|---|------|---------| +| 0001 | identity_baseline | Users, refresh tokens, verification tokens, password reset tokens | +| 0002 | auth_indexes | Performance indexes for auth lookups | +| 0003 | organizations | Organizations, members, teams, team members | +| 0004 | projects | Projects, project members, milestones, tasks, labels | +| 0005 | task_details | Task comments, activity, dependencies, task label junction | +| 0006 | github | GitHub connections, repositories, webhooks, pull requests, code reviews | +| 0007 | analytics | Notifications, activities, developer_metrics | +| 0008 | github_unique | Unique index on `github_connections(user_id)` for upsert | +| 0009 | chat | `chat_messages` table with cursor-pagination support | +| 0010 | ai_vectors | pgvector extension, `ai_document_chunks` with HNSW index | +| 0011 | ai_analyzer | Extended `ai_analyses.type` CHECK to include `analyzer` | +| 0012 | ai_assistant | `ai_conversations.repository_id`, cascade delete, index | + +### Creating a New Migration + +```bash +npm run db:migrate:create -- add_feature_x +``` + +This creates `NNNN_add_feature_x.js` with `up`/`down` stubs. + +### Migration Best Practices + +1. **Expand → Migrate → Contract** for zero-downtime deployments +2. Every migration must have an idempotent rollback +3. Document any data transformation needed for backfills +4. Add indexes CONCURRENTLY in production (not inside transactions) + +--- + +## Seed Data + +`npm run db:seed` creates an idempotent demo dataset: + +### Users + +| Email | Role | Org | +|-------|------|-----| +| `alaa@devforge.test` | Owner | DevForge Inc. | +| `jordan@devforge.test` | Admin | Acme Labs | +| `sam@devforge.test` | Developer | DevForge Inc. | + +Password for all: `DevForgeDev123!` + +### Organizations & Projects + +- **DevForge Inc.** — `DF` project (kanban board with tasks, milestones, + labels, GitHub repos) +- **Acme Labs** — `AL` project + +### Historical Data + +Enriched seed includes: 8 historical pull requests, 6 done tasks with +estimates, completed code reviews, GitHub logins — dashboards are +meaningful on a fresh database. + +--- + +## Redis Keys (Non-Persistent) + +| Purpose | Key Pattern | TTL | +|---------|-------------|-----| +| Rate limiting | `rl:{scope}:{id}:{route}` | Sliding window | +| Session cache | `session:{userId}` | 24h | +| Presence | `presence:{orgId}:{userId}` | 60s (heartbeat) | +| Socket pub/sub | `socket.io:*` | — | +| Repo metadata cache | `gh:repo:{repoId}` | 5m | + +--- + +## Performance Indexes + +Every foreign key has an index. Additional hot-path indexes: + +| Table | Index | Purpose | +|-------|-------|---------| +| `tasks` | `(project_id, status)` | Kanban board queries | +| `tasks` | `(project_id, created_at)` | Chronological listing | +| `notifications` | `(user_id, read_at)` | Unread badge | +| `developer_metrics` | `(organization_id, period)` | Analytics queries | +| `ai_document_chunks` | HNSW `vector_cosine_ops` | Vector similarity search | +| `pull_requests` | `(repository_id, state)` | PR listing | + +--- + +## pgvector Extension + +Enabled via migration 0010. Used for AI embeddings: + +```sql +CREATE EXTENSION IF NOT EXISTS vector; + +CREATE TABLE ai_document_chunks ( + id UUID PRIMARY KEY, + repository_id UUID NOT NULL, + path TEXT NOT NULL, + content TEXT NOT NULL, + language TEXT, + embedding vector(1536), + chunk_index INTEGER, + created_at TIMESTAMPTZ DEFAULT now() +); + +CREATE INDEX ON ai_document_chunks + USING hnsw (embedding vector_cosine_ops); +``` + +Hybrid retrieval: combines pgvector cosine similarity with PostgreSQL +full-text search for RAG context assembly. + +--- + +*See also: [data model](../architecture/data-model.md) · +[architecture diagrams](../architecture/diagrams.md)*