Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,14 @@ All notable changes will be documented here. The project intends to follow seman

### Added

- Bounded unsigned byte and endian-aware integer reads.
- Checked binary range validation with typed library errors.
- No-copy normalization for `Uint8Array` and `ArrayBuffer` inputs.
- Signature-based JPEG, PNG, and WebP format detection.
- Explicit format-only inspection reports with input-size enforcement.
- Binary boundary, sliced-view, format, and malformed-input tests.
- Bounded JPEG marker and length-prefixed segment traversal.
- Standalone, fill-byte, restart-marker, EOI, and multi-scan handling.
- JPEG APP and COM container classification.
- EXIF, standard/extended XMP, ICC, Photoshop/IPTC, JFIF/JFXX, and Adobe presence detection.
- Structured malformed, truncation, trailing-data, and segment-limit diagnostics.
- Normalized JPEG metadata-container entries and complete/partial container status.
- Bounded binary reader, no-copy input normalization, and JPEG/PNG/WebP format detection.
- Binary boundary, sliced-view, format, malformed-input, and JPEG container tests.

### Foundation

Expand Down
24 changes: 16 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,31 @@
# secure-metadata

`secure-metadata` is a pre-release TypeScript library for deterministic, security-conscious inspection, cleaning, and verification of metadata in binary image formats. It is being built for privacy-first, entirely local use with no analytics, telemetry, network access, runtime CDN, or pixel decoding.
`secure-metadata` is a pre-release TypeScript library for deterministic, security-conscious inspection, cleaning, and verification of metadata in binary image formats. It is built for privacy-first, entirely local use with no analytics, telemetry, network access, runtime CDN, or pixel decoding.

## Development status

The current implementation provides a bounded binary input core and signature-based JPEG, PNG, and WebP format detection. `inspectMetadata` returns an explicit `format-only` report; metadata decoding, cleaning, and verification are not implemented.
Current implementation:

- bounded binary input and endian-aware read core;
- JPEG, PNG, and WebP signature detection;
- bounded JPEG marker and segment traversal;
- JPEG entropy-scan skipping without image decoding;
- JPEG EXIF, XMP, extended XMP, ICC, Photoshop/IPTC, and comment container-presence detection.

Not implemented: TIFF/EXIF or GPS field decoding, XML/IPTC/ICC payload decoding, PNG/WebP container parsing, metadata cleaning, and verification.

## Format status

JPEG is detected from `FF D8`, PNG from its complete eight-byte signature, and WebP from `RIFF` plus `WEBP` identifiers. Detection identifies a likely container only. It does not yet validate JPEG segments, PNG chunks, WebP RIFF sizes or chunks, or any metadata. See [format support](docs/format-support.md).
JPEG reports can be `container-inspected` or `container-partial`. PNG and WebP remain `format-only`. See [format support](docs/format-support.md) for the precise matrix.

A detected or traversed container is not necessarily a decodable image. The JPEG parser validates marker and segment boundaries, not quantization, Huffman, frame, scan-header, or entropy semantics.

## Installation

The package is not published. Installation instructions will be added for the first pre-release.

## Public API

The top-level API is deliberately small:

```ts
import {
cleanMetadata,
Expand All @@ -26,17 +34,17 @@ import {
} from "secure-metadata";
```

`inspectMetadata` currently normalizes `Uint8Array | ArrayBuffer` input without copying it, enforces `maxInputBytes`, detects the container signature, and returns an empty-entry report marked `inspectionStatus: "format-only"`. That status means metadata has not been decoded; it does not claim metadata is absent.
`inspectMetadata` accepts `Uint8Array | ArrayBuffer`, enforces relevant parse limits, and returns a deterministic report. For JPEG it inventories the container and emits one normalized entry per recognized privacy/color metadata container. Entries identify container presence only; payload values are not decoded.

`cleanMetadata` and `verifyMetadata` still throw a typed `NotImplementedError`. Node.js `Buffer` values work structurally as `Uint8Array` but are not part of the public contract.

## Security philosophy

Every byte is untrusted. Binary reads use centrally checked ranges, parser input views retain their original boundaries, and malformed or tiny inputs are ordinary data. Cleaning will preserve unknown structures and ICC/color information by default, and cleaner output will be independently inspectable. See the [security model](docs/security-model.md), [architecture](docs/architecture.md), and [cleaning policy](docs/cleaning-policy.md).
Every byte is untrusted. Binary reads use centrally checked ranges, parser input views retain their original boundaries, traversal is hard bounded, and malformed or tiny inputs are ordinary data. Unknown APP segments remain unknown and should be preserved by future cleaning. See the [security model](docs/security-model.md), [architecture](docs/architecture.md), and [cleaning policy](docs/cleaning-policy.md).

## Non-goals

The library does not perform image decoding or encoding, visual redaction, pixel-content privacy analysis, steganography detection, or malware scanning. Absence of decoded metadata is never proof that an image contains no private information.
The library does not perform image decoding or encoding, visual redaction, pixel-content privacy analysis, steganography detection, or malware scanning. Absence of recognized metadata containers is never proof that an image contains no private information.

## Secure Tools ecosystem

Expand Down
41 changes: 21 additions & 20 deletions docs/architecture.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Architecture

`secure-metadata` is organized as a side-effect-free binary library. Its planned data flow is:
`secure-metadata` is organized as a side-effect-free binary library. Its current and planned flow is:

```text
Input bytes implemented
Expand All @@ -9,35 +9,36 @@ Safe binary view / bounded reads implemented
Format detection implemented
Container parser planned
JPEG container parser implemented for JPEG
Metadata decoder planned
Metadata container classification implemented for JPEG
Metadata normalization/classification planned
Metadata payload decoder planned
Inspector / policy engine planned
Normalization / field classification planned
Cleaner planned
Policy engine and cleaner planned
Output bytes planned
Re-inspection / verification planned
Output re-inspection / verification planned
```

## Binary core

All future format parsers build on `src/core/binary`. Input normalization returns the caller's exact `Uint8Array` view or creates a no-copy view over an `ArrayBuffer`. `ByteReader` validates offsets and lengths as non-negative safe integers and checks remaining capacity with subtraction before every read. It provides bounded unsigned 8-, 16-, and 32-bit reads in both endian orders, subarray views, and allocation-free signature matching.
Input normalization returns the caller's exact `Uint8Array` view or creates a no-copy view over an `ArrayBuffer`. `ByteReader` validates offsets and lengths as non-negative safe integers and checks remaining capacity with subtraction before every read. It provides bounded unsigned 8-, 16-, and 32-bit reads, subarray views, and allocation-free signature matching.

## JPEG container layer

The iterative JPEG parser validates SOI and walks markers using the binary core. A central marker model distinguishes SOI, EOI, TEM, RST0–RST7, APP0–APP15, COM, SOS, common image-structure markers, and length-prefixed unknown markers. Repeated `FF` fill bytes are collapsed to one marker; declared lengths include their two-byte length field and must fit fully before offsets advance.

After SOS, the parser scans rather than decodes entropy data. `FF 00` remains stuffed data, restart markers are recorded without terminating the scan, and the next real marker resumes normal traversal. This supports multiple scans. Every marker, including SOI, EOI, SOS, and restarts, counts toward `maxSegments`.

`ByteReader` is an internal implementation primitive, not part of the stable package exports. Its `DataView` is constrained to the input view's `byteOffset` and `byteLength`, and accesses occur only after project-owned bounds validation.
APP signatures are checked within segment payload boundaries without retaining payload copies. EXIF, standard/extended XMP, ICC, Photoshop/IPTC, JFIF/JFXX, Adobe, and unknown classifications remain container-level observations.

## Layer boundaries
## Inspection status

- **Format detection** identifies PNG, JPEG, and WebP signatures in that explicit order. It does not imply structural validity.
- **Container parsing** will identify and bound JPEG segments, PNG chunks, or WebP RIFF chunks without decoding pixels.
- **Metadata decoding** will interpret known metadata payloads. EXIF/TIFF will be one shared decoder reused by JPEG, PNG, and WebP.
- **Normalization and classification** maps format-specific fields to stable namespaces and semantic categories.
- **Privacy relevance** is an independent description of whether an entry can concern privacy. It is not a contextual risk score.
- **Cleaning policy** decides which proven structures to remove while preserving required, color, rendering, image-payload, and unknown data by default.
- **Verification** independently re-inspects cleaner output and compares it with an explicit expectation.
- `format-only`: a signature was detected; no container parser ran. Currently PNG, WebP, unknown, and short arbitrary inputs.
- `container-inspected`: JPEG traversal reached EOI safely.
- `container-partial`: JPEG identity is known, but traversal stopped on a structural error, truncation, or limit.
- `metadata-inspected`: reserved for future payload decoders.

`inspectMetadata` currently stops after detection and returns `inspectionStatus: "format-only"`. Empty entries therefore mean “not decoded,” not “confirmed absent.” Public APIs remain free of filesystem, network, browser-global, and other environmental side effects.
An empty entry list means no supported metadata container was recognized during the completed portion of traversal. It does not prove metadata or private information is absent.
42 changes: 23 additions & 19 deletions docs/format-support.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,33 @@
# Format Support

Sprint 1 implements deterministic container signature detection only. Planned parser implementation priority remains:

1. JPEG
2. WebP
3. PNG

Detection order is explicitly PNG, JPEG, WebP, then unknown. The supported signatures are distinct, so the ordering does not create heuristic ambiguity.
| Capability | JPEG | PNG | WebP |
| --------------------------- | --------------------------------- | --------- | --------- |
| Signature detection | Supported | Supported | Supported |
| Bounded container traversal | Supported | Not yet | Not yet |
| APP/COM classification | Supported | N/A | N/A |
| EXIF container detection | Supported | Not yet | Not yet |
| XMP container detection | Supported, including extended XMP | Not yet | Not yet |
| ICC container detection | Supported | Not yet | Not yet |
| IPTC/Photoshop detection | Supported | Not yet | Not yet |
| Metadata field decoding | Not yet | Not yet | Not yet |
| Cleaning | Not yet | Not yet | Not yet |

## JPEG

**Implemented:** detection when the first two bytes are `FF D8`.

This does not require an EOI marker and does not validate or traverse markers or segments. APP segments, EXIF, shared TIFF IFD and GPS decoding, XMP, IPTC, comments, and ICC distinctions remain future work. Malformed trailing bytes do not change a matching Sprint 1 signature classification.

## WebP

**Implemented:** detection of `RIFF` at offset 0 and `WEBP` at offset 8, requiring at least 12 bytes.
JPEG detection requires `FF D8`. Container inspection validates marker boundaries and two-byte big-endian declared lengths, recognizes standalone markers and fill bytes, stops at EOI, and reports trailing bytes. SOS headers are traversed, while entropy-coded bytes are skipped without decoding; `FF 00`, RST0–RST7, and multiple scans are handled structurally.

The four RIFF size bytes are deliberately ignored. RIFF size validation, chunk traversal, EXIF, XMP, ICCP, image and animation payload distinctions, and VP8X consistency handling remain future work.
Payload signatures identify:

## PNG
- APP0 `JFIF\0` and `JFXX\0` as technical container data;
- APP1 `Exif\0\0` as EXIF;
- APP1 standard and extended Adobe XMP identifiers as XMP;
- APP2 `ICC_PROFILE\0` as ICC;
- APP13 `Photoshop 3.0\0` as Photoshop/IPTC;
- APP14 `Adobe` as rendering/container data;
- COM as comment metadata.

**Implemented:** detection of the complete eight-byte PNG signature `89 50 4E 47 0D 0A 1A 0A`.
Unknown APP payloads remain unknown. No TIFF, EXIF, XMP XML, ICC, IPTC, thumbnail, frame, Huffman, quantization, or entropy payload is decoded.

A truncated or corrupted signature is unknown. IHDR and chunk structure are not inspected. Textual metadata, eXIf, XMP, ICC and color chunks, privacy-relevant ancillary chunks, CRC checking, and compressed metadata remain future work.
## PNG and WebP

`inspectionStatus: "format-only"` records this boundary in API results. A detected signature is not a claim that a file is structurally valid or that its metadata has been inspected.
PNG requires its complete eight-byte signature. WebP requires `RIFF` at offset 0 and `WEBP` at offset 8. Their chunk structures, sizes, CRCs, metadata, and image payloads are not yet parsed.
22 changes: 16 additions & 6 deletions docs/security-model.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Security Model

Binary metadata parsing processes attacker-controlled structure, sizes, offsets, encodings, and nesting. The project therefore treats malformed files, parser panics, excessive allocation or traversal, and incorrect offset arithmetic as security concerns.
Binary metadata parsing processes attacker-controlled structure, sizes, offsets, encodings, and nesting. Malformed files, parser crashes, excessive allocation or traversal, and incorrect offset arithmetic are security concerns.

## Invariants

Expand All @@ -21,15 +21,25 @@ Binary metadata parsing processes attacker-controlled structure, sizes, offsets,

## Bounded binary reads

Offsets and lengths must be non-negative safe integers. Ranges are checked with `length <= inputLength - offset`, avoiding overflow-prone addition during validation. Invalid offsets, invalid lengths, and out-of-bounds ranges throw typed library errors before `DataView` access. Signature mismatches and insufficient signature bytes return `false` rather than throwing.
Offsets and lengths must be non-negative safe integers. Ranges use `length <= inputLength - offset`, avoiding overflow-prone addition during validation. Invalid offsets, lengths, and ranges throw typed library errors before `DataView` access. A supplied `Uint8Array` retains its exact offset and length; `ArrayBuffer` normalization creates a no-copy byte view. Inspection never writes through either representation.

Normalization does not copy whole inputs. A supplied `Uint8Array` retains its exact offset and length, so bytes elsewhere in its backing buffer are inaccessible to the reader. An `ArrayBuffer` receives a no-copy byte view. The inspector never writes through either representation.
## JPEG-specific properties

## Hard limits
- Marker reads and fill-byte scans remain within the supplied input view.
- Every recorded marker, including restart markers inside scans, counts toward `maxSegments`.
- A declared segment length must be at least two and fit completely before subtraction or offset advancement.
- APP signatures must fit within their segment payload and cannot match across segment boundaries.
- Entropy-coded scan data is traversed but never decoded or copied.
- `FF 00` stuffing remains data; RST0–RST7 do not terminate a scan.
- Normal parsing resumes at non-stuffed, non-restart markers, allowing multiple SOS scans.
- EOI stops traversal; trailing bytes produce a warning rather than being parsed as JPEG.
- Malformed and truncated JPEGs return bounded structured diagnostics instead of uncontrolled native bounds exceptions.

All parser loops are iterative. Each successful branch advances its cursor or returns, which prevents non-progress cycles on hostile fill, scan, or marker data.

`inspectMetadata` enforces the effective `maxInputBytes` before detection and allocation-intensive parsing. Other default limits remain reserved for the parsers that will use them. The defaults are exported as `DEFAULT_PARSE_LIMITS`; they are conservative safeguards, not permanent API guarantees, and may evolve during `0.x` development.
## Hard limits

Limits complement bounds checks; they do not replace them. Future parsers must fail safely or produce bounded diagnostics rather than crash on malformed input.
`inspectMetadata` enforces `maxInputBytes` before parsing. JPEG traversal enforces `maxSegments`; unused limits remain reserved for their future parsers. Defaults are conservative safeguards rather than permanent `0.x` API guarantees.

## Environment and dependencies

Expand Down
14 changes: 13 additions & 1 deletion src/core/diagnostics.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,19 @@
export type DiagnosticSeverity = "warning" | "error";

export type DiagnosticCode =
"NOT_IMPLEMENTED" | "INPUT_LIMIT_EXCEEDED" | "MALFORMED_INPUT";
| "NOT_IMPLEMENTED"
| "INPUT_LIMIT_EXCEEDED"
| "MALFORMED_INPUT"
| "JPEG_INVALID_SOI"
| "JPEG_INVALID_MARKER"
| "JPEG_TRUNCATED_MARKER"
| "JPEG_TRUNCATED_SEGMENT_LENGTH"
| "JPEG_INVALID_SEGMENT_LENGTH"
| "JPEG_TRUNCATED_SEGMENT"
| "JPEG_TRUNCATED_SCAN"
| "JPEG_MISSING_EOI"
| "JPEG_SEGMENT_LIMIT_EXCEEDED"
| "JPEG_TRAILING_DATA";

export interface Diagnostic {
readonly severity: DiagnosticSeverity;
Expand Down
6 changes: 5 additions & 1 deletion src/core/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,11 @@ export interface InspectOptions {
readonly limits?: Partial<ParseLimits>;
}

export type InspectionStatus = "format-only" | "metadata-inspected";
export type InspectionStatus =
| "format-only"
| "container-inspected"
| "container-partial"
| "metadata-inspected";

export interface MetadataReport {
readonly format: ImageFormat;
Expand Down
Loading
Loading