Skip to content

docs: simplify documentation and focus around cbq #105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
noamteyssier merged 1 commit into from
Aug 26, 2026
Merged

docs: simplify documentation and focus around cbq #105

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Large cleanup of internal and public API.
- Runtime dependencies removed (`byteorder`, `num_cpus`, `auto_impl`, and `memchr`).
- Deprecated items removed from the public API.
- Documentation overhaul: docs now center on CBQ as the recommended variant (VBQ is documented as superseded), examples use CBQ throughout, and docstrings are heavily simplified across the crate.

### Removed

Expand Down
24 changes: 11 additions & 13 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# BINSEQ Format Specification
# BINSEQ

[![MIT licensed](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE.md)
![actions status](https://github.com/arcinstitute/binseq/workflows/CI/badge.svg)
Expand All @@ -7,25 +7,23 @@

## Overview

BINSEQ is a binary file format family designed for efficient storage and processing of DNA sequences.
They make use of two-bit encoding for nucleotides and are optimized for high-performance parallel processing.
BINSEQ is a binary file format family for efficient storage and processing of DNA sequences.
It uses two-bit encoding for nucleotides and is optimized for high-performance parallel processing.

BINSEQ has three variants:
The recommended variant is **CBQ** (`*.cbq`): a columnar, block-compressed format for variable-length records with optional quality scores and headers.
It is lossless by default (native `N` support), compresses well, and decodes fast.
For details on its structure see the [documentation](https://docs.rs/binseq/latest/binseq/cbq/).

1. **BQ**: (`*.bq`) files are for _fixed-length_ records **without** quality scores.
2. **VBQ**: (`*.vbq`) files are for _variable-length_ records **with optional** quality scores and headers.
3. **CBQ**: (`*.cbq`) files are for _columnar variable-length_ records **with optional** quality scores and headers.
Two earlier variants remain supported:

All variants support both single and paired sequences.
- **BQ** (`*.bq`): fixed-length records without quality scores. Minimal and fast for uniform reads.
- **VBQ** (`*.vbq`): variable-length records with optional quality scores and headers. Superseded by CBQ, which improves on its compression and decoding speed; new projects should use CBQ.

**Note:** For most use cases, the newest variant _CBQ_ is recommended due to its flexibility, storage efficiency, and decoding speed.
It supersedes _VBQ_ in terms of performance and storage efficiency, at a small cost in encoding speed.
VBQ will still be supported but newer projects should consider using _CBQ_ instead.
For information on the structure of _CBQ_ files, see the [documentation](https://docs.rs/binseq/latest/binseq/cbq/).
All variants support both single and paired sequences.

## Getting Started

This is a **library** for reading and writing BINSEQ files, for a **command-line interface** see [bqtools](https://github.com/arcinstitute/bqtools).
This is a **library** for reading and writing BINSEQ files; for a **command-line interface** see [bqtools](https://github.com/arcinstitute/bqtools).

To get started please refer to our [documentation](https://docs.rs/binseq/latest/binseq/).
For example programs which make use of the library check out our [examples directory](https://github.com/arcinstitute/binseq/tree/main/examples).
Expand Down
145 changes: 18 additions & 127 deletions src/bq/header.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,4 @@
//! Header module for the binseq library
//!
//! This module provides the header structure and functionality for binary sequence files.
//! The header contains metadata about the binary sequence data, including format version,
//! sequence length, and other information necessary for proper interpretation of the data.
//! Fixed-size file header for BQ files.

use bitnuc_deprec::BitSize;
use std::io::{Read, Write};
Expand All @@ -12,30 +8,17 @@ use crate::{
utils::read_u32_le,
};

/// Current magic number: "BSEQ" in ASCII (in little-endian byte order)
///
/// This is used to identify binary sequence files and verify file integrity.
#[allow(clippy::unreadable_literal)]
const MAGIC: u32 = 0x51455342;
/// The magic bytes at the start of a BQ file on disk.
pub const FILE_MAGIC: [u8; 4] = *b"BSEQ";
const MAGIC: u32 = u32::from_le_bytes(FILE_MAGIC);

/// The magic bytes as they appear at the start of a BQ file on disk.
///
/// Used to identify BQ files by content rather than by file extension.
pub const FILE_MAGIC: [u8; 4] = MAGIC.to_le_bytes();

/// Current format version of the binary sequence file format
///
/// This version number allows for future format changes while maintaining backward compatibility.
/// Current format version
const FORMAT: u8 = 1;

/// Size of the header in bytes
///
/// The header has a fixed size to ensure consistent reading and writing of binary sequence files.
pub const SIZE_HEADER: usize = 32;

/// Reserved bytes in the header
///
/// These bytes are reserved for future use and should be set to a consistent value.
/// Reserved bytes in the header (future use)
pub const RESERVED: [u8; 17] = [42; 17];

#[derive(Debug, Clone, Copy)]
Expand Down Expand Up @@ -98,48 +81,28 @@ impl FileHeaderBuilder {
}
}

/// Header structure for binary sequence files
///
/// The `FileHeader` contains metadata about the binary sequence data stored in a file,
/// including format information, sequence lengths, and space for future extensions.
///
/// The total size of this structure is 32 bytes, with a fixed layout to ensure
/// consistent reading and writing across different platforms.
/// Fixed 32-byte header for BQ files.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct FileHeader {
/// Magic number to identify the file format
///
/// 4 bytes
/// Magic number identifying the file format (4 bytes)
pub magic: u32,

/// Version of the file format
///
/// 1 byte
/// Format version (1 byte)
pub format: u8,

/// Length of all sequences in the file
///
/// 4 bytes
/// Primary sequence length (4 bytes)
pub slen: u32,

/// Length of secondary sequences in the file
///
/// 4 bytes
/// Secondary sequence length (4 bytes)
pub xlen: u32,

/// Number of bits per nucleotide (currently 2 or 4)
///
/// 1 byte
/// Bits per nucleotide, 2 or 4 (1 byte)
pub bits: BitSize,

/// All records have a flag attribute
///
/// 1 byte
/// Whether all records carry a flag attribute (1 byte)
pub flags: bool,

/// Reserve remaining bytes for future use
///
/// 17 bytes
/// Reserved for future use (17 bytes)
pub reserved: [u8; 17],
}
impl FileHeader {
Expand All @@ -149,26 +112,7 @@ impl FileHeader {
self.xlen > 0
}

/// Parses a header from a fixed-size byte array
///
/// This method validates the magic number and format version before constructing
/// a header instance. If validation fails, appropriate errors are returned.
///
/// # Arguments
///
/// * `buffer` - A byte array of exactly `SIZE_HEADER` bytes containing the header data
///
/// # Returns
///
/// * `Ok(FileHeader)` - A valid header parsed from the buffer
/// * `Err(Error)` - If the buffer contains invalid header data
///
/// # Errors
///
/// Returns an error if:
/// * The magic number is incorrect
/// * The format version is unsupported
/// * The reserved bytes are invalid
/// Parses and validates a header from a fixed-size byte array
pub fn from_bytes(buffer: &[u8; SIZE_HEADER]) -> Result<Self> {
let magic = read_u32_le(&buffer[0..4]);
if magic != MAGIC {
Expand Down Expand Up @@ -200,26 +144,7 @@ impl FileHeader {
})
}

/// Parses a header from an arbitrarily sized buffer
///
/// This method extracts the header from the beginning of a buffer that may be larger
/// than the header size. It checks that the buffer is at least as large as the header
/// before attempting to parse it.
///
/// # Arguments
///
/// * `buffer` - A byte slice containing at least `SIZE_HEADER` bytes
///
/// # Returns
///
/// * `Ok(FileHeader)` - A valid header parsed from the buffer
/// * `Err(Error)` - If the buffer is too small or contains invalid header data
///
/// # Errors
///
/// Returns an error if:
/// * The buffer is smaller than `SIZE_HEADER`
/// * The header data is invalid (see `from_bytes` for validation details)
/// Parses a header from the first `SIZE_HEADER` bytes of a buffer
pub fn from_buffer(buffer: &[u8]) -> Result<Self> {
let mut bytes = [0u8; SIZE_HEADER];
if buffer.len() < SIZE_HEADER {
Expand All @@ -229,23 +154,7 @@ impl FileHeader {
Self::from_bytes(&bytes)
}

/// Writes the header to a writer
///
/// This method serializes the header to its binary representation and writes it
/// to the provided writer.
///
/// # Arguments
///
/// * `writer` - Any type that implements the `Write` trait
///
/// # Returns
///
/// * `Ok(())` - If the header was successfully written
/// * `Err(Error)` - If writing to the writer failed
///
/// # Errors
///
/// Returns an error if writing to the writer fails (typically an I/O error).
/// Serializes the header and writes it to a writer
pub fn write_bytes<W: Write>(&self, writer: &mut W) -> Result<()> {
let mut buffer = [0u8; SIZE_HEADER];
buffer[0..4].copy_from_slice(&self.magic.to_le_bytes());
Expand All @@ -259,25 +168,7 @@ impl FileHeader {
Ok(())
}

/// Reads a header from a reader
///
/// This method reads exactly `SIZE_HEADER` bytes from the provided reader and
/// parses them into a header structure.
///
/// # Arguments
///
/// * `reader` - Any type that implements the `Read` trait
///
/// # Returns
///
/// * `Ok(FileHeader)` - A valid header read from the reader
/// * `Err(Error)` - If reading from the reader failed or the header data is invalid
///
/// # Errors
///
/// Returns an error if:
/// * Reading from the reader fails (typically an I/O error)
/// * The header data is invalid (see `from_bytes` for validation details)
/// Reads and parses a header from a reader
pub fn from_reader<R: Read>(reader: &mut R) -> Result<Self> {
let mut buffer = [0u8; SIZE_HEADER];
reader.read_exact(&mut buffer)?;
Expand Down
Loading