diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..055d6c2 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,27 @@ +# FRS Project Documentation + +Welcome to the documentation for **FRS**, a Flask-based face recognition system that captures webcam video, detects faces, recognizes registered people, and lets users register newly detected faces from a browser interface. + +## Documentation Index + +- [Project Overview](./project-overview.md) - Purpose, features, repository structure, and high-level behavior. +- [Architecture](./architecture.md) - Component layout, threading model, data flow, and runtime lifecycle. +- [Backend Reference](./backend-reference.md) - Flask routes, recognition pipeline, constants, globals, and persistence behavior. +- [Frontend Reference](./frontend-reference.md) - HTML, CSS, JavaScript, UI behavior, and browser-to-server interactions. +- [Setup and Running](./setup-and-running.md) - Requirements, installation, platform notes, and run instructions. +- [Data Storage and Privacy](./data-storage-and-privacy.md) - Face encoding storage, generated files, privacy considerations, and safe handling. +- [Operations and Troubleshooting](./operations-and-troubleshooting.md) - Common runtime issues, camera problems, dependency problems, and tuning guidance. +- [Development Guide](./development-guide.md) - Code organization, contribution workflow, testing ideas, and extension points. + +## Quick Start + +```bash +python -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt +python app.py +``` + +After the server starts, open `http://localhost:5000` in a browser with access to the machine running the webcam. + +> Note: This project requires a working camera device and native dependencies used by OpenCV, dlib, and `face_recognition`. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..2c346d4 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,87 @@ +# Architecture + +## High-Level Architecture + +FRS is a single Flask application with three major layers: + +1. **Camera and recognition layer** in `app.py`. +2. **HTTP API and streaming layer** in `app.py`. +3. **Browser interface layer** in `templates/index.html`, `static/script.js`, and `static/style.css`. + +```text +Webcam + │ + ▼ +VideoProcessor capture thread + │ stores latest frame + ▼ +VideoProcessor recognition thread + │ analyzes periodic frames + ▼ +latest detections + status + │ + ├── /video -> annotated MJPEG stream + ├── /status -> JSON status text + └── /register -> saves latest unknown face encoding +``` + +## Runtime Lifecycle + +1. Python imports `app.py`. +2. The app checks for `registered_faces.pkl`. +3. Registered face encodings are loaded if the file exists; otherwise an empty registry is created. +4. The known-face cache is initialized. +5. `VideoProcessor().start()` opens the default camera and starts two daemon threads. +6. Flask serves routes when `app.py` is executed directly. +7. On application shutdown, `video_processor.release()` attempts to stop threads and release the camera. + +## Threading Model + +The application separates capture, recognition, and streaming so the live feed remains responsive even when face recognition is slower than camera capture. + +### Capture Thread + +The capture thread continuously reads frames from OpenCV's `VideoCapture` object and stores the latest frame in memory. It targets `CAPTURE_FPS` and uses a lock to protect frame access. + +### Recognition Thread + +The recognition thread periodically copies the latest frame, downsizes it, detects faces, computes encodings, compares them to known encodings, and updates the latest detection list and status text. It targets `RECOGNITION_FPS`, which is lower than the stream frame rate to reduce CPU usage. + +### Request Threads + +Flask handles HTTP requests. Route handlers read shared state through lock-protected methods and return HTML, JSON, or streaming frame data. + +## Shared State + +| State | Purpose | Protection | +| --- | --- | --- | +| `registered_faces` | Persistent in-memory mapping of names to encodings | Refreshed through helper functions | +| `known_face_names` | Cached list of registered names for matching | `known_faces_lock` | +| `known_face_encodings` | Cached list of registered encodings for matching | `known_faces_lock` | +| `temp_face_encoding` | Latest unknown face encoding available for registration | `temp_face_lock` | +| `latest_frame` | Most recent webcam frame | `frame_lock` | +| `latest_detections` | Most recent recognition results | `detections_lock` | +| `latest_status` | Text shown by `/status` | `status_lock` | + +## Face Recognition Data Flow + +1. A full-size BGR camera frame is captured by OpenCV. +2. Recognition resizes the frame by `RECOGNITION_SCALE`. +3. The resized frame is converted from BGR to RGB for `face_recognition`. +4. Face locations are found using the HOG model. +5. Face encodings are computed for detected locations. +6. Each encoding is compared with the cached registered encodings. +7. The closest match is accepted only if its distance is less than or equal to `MATCH_TOLERANCE`. +8. Small-frame coordinates are scaled back up to the displayed frame size. +9. Detections are cached for rendering and status updates. + +## Video Streaming Flow + +The `/video` route returns a multipart MJPEG response. Each loop iteration: + +1. Copies the latest frame. +2. Draws the latest detections on top of the copied frame. +3. Encodes the frame as JPEG. +4. Yields the encoded bytes with the multipart boundary expected by browsers. + +This lets a normal `` element display a continuously updating camera stream. diff --git a/docs/backend-reference.md b/docs/backend-reference.md new file mode 100644 index 0000000..382bee1 --- /dev/null +++ b/docs/backend-reference.md @@ -0,0 +1,137 @@ +# Backend Reference + +## Entry Point + +The backend lives in `app.py`. Running `python app.py` starts the Flask development server on `0.0.0.0:5000` and initializes webcam processing. + +## Configuration Constants + +| Constant | Purpose | +| --- | --- | +| `REGISTERED_FACES_PATH` | File path used to persist registered face encodings. | +| `FRAME_WIDTH` / `FRAME_HEIGHT` | Requested webcam capture resolution. | +| `CAPTURE_FPS` | Target camera capture rate. | +| `STREAM_FPS` | Target MJPEG streaming rate. | +| `JPEG_QUALITY` | JPEG encoding quality for streamed frames. | +| `RECOGNITION_SCALE` | Downscale factor used before recognition for speed. | +| `RECOGNITION_FPS` | Target recognition loop rate. | +| `MATCH_TOLERANCE` | Maximum face distance accepted as a match. Lower values are stricter. | + +## Persistent Registry + +At startup, the application looks for `registered_faces.pkl` in the working directory. If it exists, the file is loaded with `pickle`. If it does not exist, the app starts with an empty registry. + +The registry structure is: + +```python +{ + "Person Name": { + "encoding": + } +} +``` + +When a new face is registered, the registry is written back to `registered_faces.pkl`. + +## Known-Face Cache + +`refresh_known_face_cache()` converts the registry dictionary into two lists: + +- `known_face_names` +- `known_face_encodings` + +These lists make recognition matching simpler and faster because the code can compare a detected encoding against all cached known encodings. + +## `VideoProcessor` + +`VideoProcessor` owns the webcam capture object and the background processing threads. + +### Responsibilities + +- Open and configure the camera. +- Capture frames continuously. +- Run recognition periodically. +- Store the latest frame, detections, and status. +- Produce annotated JPEG frames for the video stream. +- Release the camera on shutdown. + +### Important Methods + +| Method | Description | +| --- | --- | +| `start()` | Starts capture and recognition daemon threads. | +| `_capture_loop()` | Reads camera frames and stores the latest frame. | +| `_recognition_loop()` | Runs face recognition against the latest frame. | +| `get_frame()` | Returns a copy of the latest frame. | +| `get_detections()` | Returns the most recent detection list. | +| `get_status()` | Returns the current status text. | +| `_update_status()` | Converts detections into user-facing status text. | +| `get_jpeg_frame()` | Draws detections and returns a JPEG-encoded frame. | +| `release()` | Stops processing and releases camera resources. | + +## Face Registration + +`register_new_face(name, face_encoding)` stores the supplied encoding in `registered_faces`, refreshes the known-face cache, and serializes the registry to disk. + +The `/register` route uses `temp_face_encoding`, which is set when the rendering layer sees an unknown face in the latest detections. Registration can fail if: + +- The submitted name is empty. +- No new face encoding is currently available. + +## Recognition Pipeline + +`recognize_faces(frame)` performs the following steps: + +1. Resize the frame using `RECOGNITION_SCALE`. +2. Convert the resized frame from BGR to RGB. +3. Locate faces with `face_recognition.face_locations(..., model="hog")`. +4. Generate encodings with `face_recognition.face_encodings(...)`. +5. Compare each encoding against cached known encodings using face distance. +6. Select the closest known face with `numpy.argmin`. +7. Accept the match only when the distance is within `MATCH_TOLERANCE`. +8. Scale face coordinates back to the original frame size. +9. Return tuples of `(match, face_location, face_encoding)`. + +## Detection Rendering + +`draw_detections(frame, detections)` draws a rectangle and label for each detected face: + +- Green rectangle and registered name for known faces. +- Red rectangle and `New Face` for unknown faces. + +When an unknown face is rendered, its encoding is stored as `temp_face_encoding` so it can be registered by the form. + +## Flask Routes + +| Route | Method | Response | Purpose | +| --- | --- | --- | --- | +| `/` | `GET` | HTML | Renders the main web UI. | +| `/video` | `GET` | Multipart MJPEG stream | Streams annotated camera frames. | +| `/status` | `GET` | JSON | Returns current face recognition status. | +| `/register` | `POST` | JSON | Registers the latest unknown face with a submitted name. | + +### `/status` Response Example + +```json +{ + "status": "Recognized: Ada" +} +``` + +### `/register` Request Example + +```http +POST /register +Content-Type: application/x-www-form-urlencoded + +name=Ada +``` + +### `/register` Success Example + +```json +{ + "ok": true, + "message": "Registered Ada." +} +``` diff --git a/docs/data-storage-and-privacy.md b/docs/data-storage-and-privacy.md new file mode 100644 index 0000000..d5347ea --- /dev/null +++ b/docs/data-storage-and-privacy.md @@ -0,0 +1,57 @@ +# Data Storage and Privacy + +## What Data Is Stored + +FRS stores registered face encodings, not raw face images. A face encoding is a numerical representation generated from a detected face. These encodings are still biometric identifiers and should be handled carefully. + +## Storage Location + +Registered faces are persisted in: + +```text +registered_faces.pkl +``` + +The file is created in the process working directory when a user successfully registers a face. + +## Storage Format + +The file is serialized with Python `pickle`. The in-memory structure maps a submitted name to an encoding object: + +```python +{ + "Name": { + "encoding": encoding + } +} +``` + +## Security Considerations + +- Do not commit `registered_faces.pkl` to source control. +- Do not share the pickle file publicly. +- Restrict filesystem access to the machine running the app. +- Avoid accepting untrusted pickle files because loading pickle data can execute arbitrary code. +- Consider replacing pickle with a safer format or database layer before production use. +- Add authentication before exposing the application outside a trusted local environment. + +## Privacy Considerations + +Face encodings can identify people and should be treated as sensitive biometric data. Before using the system with real people: + +- Get clear consent. +- Explain what is stored and why. +- Provide a way to remove registered identities. +- Define a retention policy. +- Secure backups and exported data. +- Follow applicable privacy laws and organizational policies. + +## Data Deletion + +To remove all registered faces, stop the application and delete: + +```bash +rm registered_faces.pkl +``` + +To remove one person, a management function or route would need to be added because the current application only supports adding or replacing entries by name. diff --git a/docs/development-guide.md b/docs/development-guide.md new file mode 100644 index 0000000..0d8d8ae --- /dev/null +++ b/docs/development-guide.md @@ -0,0 +1,81 @@ +# Development Guide + +## Code Organization + +The project is intentionally compact: + +- `app.py` contains backend configuration, face recognition logic, streaming, registration, and Flask routes. +- `templates/index.html` contains the single rendered page. +- `static/script.js` contains browser behavior. +- `static/style.css` contains UI styling. +- `requirements.txt` lists Python dependencies. +- `docs/` contains this documentation set. + +## Local Development Workflow + +```bash +python -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt +python app.py +``` + +After making changes, manually verify: + +1. The server starts without import errors. +2. The browser loads `/`. +3. The video feed renders. +4. `/status` returns JSON. +5. Registration succeeds when an unknown face is visible. +6. `registered_faces.pkl` is created or updated. + +## Suggested Automated Checks + +This repository currently does not include a dedicated test suite. Useful future checks include: + +- Python syntax compilation with `python -m py_compile app.py`. +- Unit tests for registration and recognition helper functions with mocked encodings. +- Flask route tests using Flask's test client. +- Frontend smoke checks for expected DOM elements. +- Formatting and linting with tools such as Ruff or Black. + +## Extension Points + +### Configuration + +Move constants from `app.py` into environment variables or a config file to make deployment easier. + +### Storage + +Replace `registered_faces.pkl` with SQLite or another database if you need safer querying, deletion, migrations, or metadata. + +### Identity Management + +Add routes and UI for: + +- Listing registered people. +- Deleting a registered person. +- Updating names. +- Exporting or backing up registrations. + +### Security + +Add authentication before exposing the app beyond a trusted local environment. Registration currently accepts any form submission that can reach the server. + +### Recognition Models + +The current face location model is `hog`, which is CPU-friendly. Systems with GPU support could experiment with CNN-based detection, but this requires additional native setup and more compute. + +### API Design + +If the frontend grows, consider documenting and versioning the JSON endpoints, for example under `/api/status` and `/api/register`. + +## Contribution Guidelines + +When contributing: + +- Keep changes focused and easy to review. +- Update documentation when behavior changes. +- Avoid committing generated biometric data such as `registered_faces.pkl`. +- Test with an actual camera when changing recognition or streaming behavior. +- Be careful with threading changes and protect shared mutable state with locks. diff --git a/docs/frontend-reference.md b/docs/frontend-reference.md new file mode 100644 index 0000000..7e0c264 --- /dev/null +++ b/docs/frontend-reference.md @@ -0,0 +1,67 @@ +# Frontend Reference + +## Overview + +The frontend is intentionally lightweight. Flask renders a single HTML page, static CSS styles the interface, and a small JavaScript file handles status polling and registration form submission. + +## HTML Template + +`templates/index.html` defines the page structure: + +- Page title and stylesheet link. +- Main application shell. +- Status text area. +- MJPEG video feed displayed through an `` element. +- Registration form with a name input. +- Message area for success and error feedback. +- Script include for frontend behavior. + +The video feed uses: + +```html +Live face recognition video feed +``` + +Because `/video` returns multipart MJPEG data, the browser keeps updating the image as new frames arrive. + +## JavaScript Behavior + +`static/script.js` performs three tasks: + +1. Handles the registration form submit event. +2. Sends the entered name to `/register` as URL-encoded form data. +3. Polls `/status` every 500 milliseconds and updates the status text. + +## Registration Flow + +1. User enters a name. +2. User clicks **Register Face**. +3. JavaScript prevents normal form navigation. +4. Empty names are rejected in the browser. +5. A `POST /register` request is sent with `Content-Type: application/x-www-form-urlencoded`. +6. The JSON response controls the feedback message. +7. On success, the input field is cleared. + +## Status Polling + +`refreshStatus()` fetches `/status` with `cache: 'no-store'` so the browser does not reuse stale responses. If the request fails, the UI shows `Waiting for camera stream...`. + +## Styling + +`static/style.css` creates a centered dark interface with: + +- White text on a dark background. +- Green accent color for active status and buttons. +- A glowing green border around the video feed. +- Responsive wrapping for the registration form. +- Separate success and error message colors. + +## Browser Requirements + +The frontend should work in modern browsers that support: + +- `fetch()`. +- `URLSearchParams`. +- Multipart MJPEG display through an `` element. + +The browser does not directly access the webcam. Camera access happens on the server machine through OpenCV. diff --git a/docs/operations-and-troubleshooting.md b/docs/operations-and-troubleshooting.md new file mode 100644 index 0000000..2fb85f4 --- /dev/null +++ b/docs/operations-and-troubleshooting.md @@ -0,0 +1,97 @@ +# Operations and Troubleshooting + +## Common Startup Problems + +### `ModuleNotFoundError` + +Install dependencies: + +```bash +pip install -r requirements.txt +``` + +Make sure you are using the same Python environment that runs `app.py`. + +### `face_recognition` or `dlib` Build Failure + +`face_recognition` depends on native packages. Install platform build tools: + +- Linux: compiler, Python development headers, CMake, and related native libraries. +- Windows: Visual Studio Build Tools with Desktop development with C++. + +### Camera Does Not Open + +The app uses `cv2.VideoCapture(0)`, which selects the default camera. Check that: + +- A camera is connected. +- Another application is not already using the camera. +- The process has camera permissions. +- The correct camera index is used if multiple cameras exist. + +To use another camera, instantiate `VideoProcessor` with a different source index or video source. + +## Runtime Problems + +### Video Stream Is Blank + +Possible causes: + +- Camera frames are not being captured. +- Browser cannot reach the Flask server. +- OpenCV cannot access the camera device. +- The server process is running in an environment without camera passthrough. + +### Recognition Is Slow + +Tune these constants in `app.py`: + +- Lower `RECOGNITION_FPS` to analyze fewer frames per second. +- Lower `RECOGNITION_SCALE` for faster recognition with less detail. +- Lower `FRAME_WIDTH` and `FRAME_HEIGHT` to reduce captured frame size. + +### False Matches + +Lower `MATCH_TOLERANCE` to make matching stricter. This can reduce false positives but may increase false negatives. + +### Known Person Not Recognized + +Possible fixes: + +- Register the person again with better lighting. +- Face the camera directly during registration. +- Improve lighting and reduce motion blur. +- Raise `MATCH_TOLERANCE` slightly if matching is too strict. + +### Registration Says No New Face Detected + +The backend only registers the latest unknown face encoding. Make sure: + +- A face is visible in the stream. +- The status says a new face was detected. +- The person is not already matched as a registered face. +- You submit the form soon after the unknown face appears. + +## Performance Tuning + +| Goal | Suggested Change | Tradeoff | +| --- | --- | --- | +| Faster recognition | Decrease `RECOGNITION_FPS` | Less frequent status updates | +| Lower CPU usage | Decrease capture or stream FPS | Less smooth video | +| Better visual quality | Increase `JPEG_QUALITY` | Larger stream bandwidth | +| Stricter matching | Lower `MATCH_TOLERANCE` | More missed matches | +| More permissive matching | Raise `MATCH_TOLERANCE` | More false matches | + +## Production Readiness Checklist + +Before production use, consider adding: + +- Authentication and authorization. +- HTTPS termination. +- CSRF protection for registration. +- Safer storage than pickle. +- A way to delete registered users. +- Structured logging. +- Health checks. +- Configuration through environment variables. +- Automated tests. +- Clear privacy and retention policies. diff --git a/docs/project-overview.md b/docs/project-overview.md new file mode 100644 index 0000000..0435f0a --- /dev/null +++ b/docs/project-overview.md @@ -0,0 +1,57 @@ +# Project Overview + +## What FRS Does + +FRS is a real-time face recognition web application. It uses a local webcam to capture video, analyzes frames for faces, compares detected face encodings against previously registered encodings, and streams annotated video to a browser. + +The application can: + +- Capture live webcam frames. +- Detect faces in real time. +- Compare unknown faces with registered faces. +- Draw visual labels around recognized and unknown faces. +- Expose a browser UI for viewing the camera stream. +- Register a newly detected face by submitting a name. +- Persist registered face encodings to a local pickle file. + +## Repository Structure + +```text +FRS/ +├── app.py # Flask server, webcam processing, recognition, registration, and streaming +├── requirements.txt # Python dependencies +├── README.md # Public project summary and installation notes +├── LICENSE # MIT license +├── static/ +│ ├── script.js # Browser-side status polling and registration form handling +│ └── style.css # Browser UI styling +├── templates/ +│ └── index.html # Main Flask-rendered page +└── docs/ # Detailed project documentation +``` + +## Main Technologies + +- **Python** powers the backend application. +- **Flask** serves the web page, status API, registration API, and MJPEG video stream. +- **OpenCV** captures webcam frames, resizes images, converts color spaces, draws labels, and JPEG-encodes frames. +- **face_recognition** detects faces and produces numerical face encodings. +- **NumPy** helps select the closest face match using vector distances. +- **Pickle** stores registered face encodings in `registered_faces.pkl`. +- **HTML/CSS/JavaScript** provide the browser interface. + +## User Workflow + +1. Start the Flask application with `python app.py`. +2. The backend opens the default camera and starts background capture and recognition threads. +3. The user opens the web page in a browser. +4. The browser displays the `/video` MJPEG stream. +5. The browser polls `/status` every 500 milliseconds. +6. If a known face appears, the video overlay and status show the registered name. +7. If an unknown face appears, the video overlay shows `New Face` and the status asks for registration. +8. The user enters a name and submits the registration form. +9. The backend stores the latest unknown face encoding under that name. + +## Current Scope + +The project is designed as a local, single-process face recognition demo or lightweight application. It does not include multi-user accounts, authentication, database migrations, cloud storage, or production deployment configuration. diff --git a/docs/setup-and-running.md b/docs/setup-and-running.md new file mode 100644 index 0000000..61a0cbd --- /dev/null +++ b/docs/setup-and-running.md @@ -0,0 +1,75 @@ +# Setup and Running + +## Requirements + +- Python 3.11 is recommended by the project README. +- A working webcam connected to the machine running the Flask app. +- Native build tools required by `face_recognition` and its dependencies. +- Python packages listed in `requirements.txt`: + - `cmake` + - `opencv-python` + - `face-recognition` + - `numpy` + - `Pillow` + - `flask` + +## Recommended Local Setup + +```bash +python -m venv .venv +source .venv/bin/activate +pip install --upgrade pip +pip install -r requirements.txt +python app.py +``` + +Open the application at: + +```text +http://localhost:5000 +``` + +If you run the server on another machine in the same network, open `http://:5000`. + +## Linux Notes + +Some Linux systems need OpenCV runtime libraries. If camera display or import fails with missing OpenGL-related libraries, install the appropriate system package for your distribution. On Debian or Ubuntu-based systems, this is commonly: + +```bash +sudo apt install libgl1-mesa-glx +``` + +Some newer distributions provide the package as `libgl1` instead. + +## Windows Notes + +The `face_recognition` package depends on native components. Windows users usually need: + +- Python 3.11 64-bit. +- Visual Studio Build Tools. +- The **Desktop development with C++** workload. +- A camera available to desktop applications. + +## Running the App + +Use: + +```bash +python app.py +``` + +The app starts Flask on `0.0.0.0:5000`, which means it listens on all network interfaces. For local use, browse to `http://localhost:5000`. + +## Generated Files + +The app may generate: + +```text +registered_faces.pkl +``` + +This file stores registered face encodings and should be treated as sensitive biometric data. + +## Stopping the App + +Press `Ctrl+C` in the terminal. The application attempts to release the camera in the `finally` block when running as the main script.