Skip to content

refactor: replace SQLite with recoverable JSON storage - #104

Closed
AntonyGarand wants to merge 949 commits into
masterfrom
refactor/json-only-storage
Closed

AntonyGarand wants to merge 949 commits into
masterfrom
refactor/json-only-storage

Conversation

@AntonyGarand

@AntonyGarand AntonyGarand commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Why

The Plugin Hub standard build cannot bundle the SQLite driver. SQLite was never released, so existing users need a safe migration from legacy JSON only. The current JSON path also leaves saves optional and rewrites entire account files, with weak protection against interrupted writes and competing clients.

Summary

  • Remove SQLite, backend switching and the driver. Use RuneLite-provided Gson and Java 11 for one versioned JSON store: a checkpoint, its previous copy and a JSON Lines transaction journal.
  • Migrate legacy accounts and account-wide settings once, preserving source files. Retain all trades, recipe snapshots, favorites, slot state, large prices and timestamp precision.
  • Save pending changes every second on an ordered I/O worker. Append changed records, acknowledge forced writes, retry failures and drain pending changes at shutdown. The sidebar shows Saved, Saving… or Needs attention.
  • Coordinate clients with file locking and account baseline checks. Preserve conflicting local captures for recovery and prevent stale reloads from overwriting local edits.
  • Move startup parsing/I/O off the client thread, avoid history traversal for session ticks, and sort account-wide merged histories once per item.
  • Include build=standard and the CI work from build: add standard Plugin Hub packaging and CI feedback #103: Java 11 tests plus the official RuneLite packager, with job summaries and downloadable logs/packages.

Storage format, recovery procedure and the seven-issue mapping.

Validation

  • ./gradlew clean build --no-daemon --console=plain, then ./gradlew build --no-daemon --console=plain after review fixes, on Java 11: final suite passed all 186 tests with no failures or skips.
  • Compiled production sources and packaged the JAR using the pinned Plugin Hub standard build template and RuneLite 1.13.0.
  • Synthetic 50,000-trade / 200-recipe history reloads in full; adding a trade appends under 2 KiB. Tests also cover migration parity, corrupt checkpoints/journals, interrupted initialization, concurrent clients, dirty retries, deletion, shutdown and stale refreshes.
  • GitHub Actions: Java 11 build/tests and official RuneLite Plugin Hub packaging both passed on 048ef77. Logs, test reports and the packaged plugin are available in the run artifacts.
  • Independent review passed after fixing active-account deletion, view-reset and batch-deletion selection regressions.

Notes

All history remains in memory as requested; there is no 100-trade limit, pagination or disk query index. Hydration and full-account change capture still involve client-thread work. Same-account conflicts are reported and preserved for manual reconciliation rather than automatically merged. Uncommitted/queued changes remain vulnerable to abrupt termination; directory-sync guarantees depend on the filesystem.

This draft includes #103 and leaves that PR open. No SQLite import is included because SQLite was never shipped. Legacy files stop updating after migration; older plugin versions will not see new trades. Tests use synthetic data, not player files. In-client manual validation remains for maintainer review.

🤖 cb-ship:created v1 skill@1.0.13

Agent session: codex resume 01a0e966-8bd5-75a2-a5e4-879d7b59a625 · Model: gpt-6-astra · Reasoning: ultra

The item and recipe sort methods ran their key computation inside the
comparator lambda: every one of the O(n log n) comparisons re-fetched
the recipe-adjusted interval view and re-ran the full flip pairing
(combineToFlips) for items, or re-streamed every flip of a group (and
every partial offer of every flip for profit/expense) for recipes. On
large datasets (2k+ items, hundreds of offers each) switching the sort
method took seconds.

Both sorts now precompute each element's key once (decorate-sort-
undecorate via IdentityHashMap lookups) and sort by the cached value.
The recipe sort also computes the interval-filtered flip list once per
group and reuses it for the missing-offer filter. Order results are
unchanged; complexity drops from O(n log n * H) to O(n * H + n log n).
DataHandler: a failed format-migration save (backup creation or the
write itself) was handled identically to a failed load/prepare - the
valid, fully loaded account was replaced with an empty one and its JSON
saves were disabled. The migration save is now a separate concern
(persistVersionMigration): a failure there keeps the loaded account,
logs, and marks it dirty so a later save retries. protectAccount and
the empty-account replacement apply only when loading or preparing the
data actually fails.

WikiDataFetcherJob: completeRequest notified subscribers while holding
the job monitor, delaying the scheduler thread and world-switch fetch
attempts on the same monitor (deadlock risk if a subscriber takes a
lock held while fetching). The completion timestamp and a subscriber
snapshot are now captured inside the monitor and the subscribers are
notified after releasing it.
Updates the wiki endpoints to v2 (verified against the live API and the
wiki API docs):

- /osrs/latest moves to v2. The docs state prices may exceed what 32 bit
  integers hold, unseen trade sides are null, and average prices may
  include decimals. WikiItemMargins prices are now Long, parsed
  leniently (numbers, decimals, string-encoded values, nulls) via
  LenientLongAdapter; saturating capped accessors serve the int-typed
  consumers (offer prices and GE listings cannot exceed max cash, so
  clamping preserves comparison semantics).
- /osrs/timeseries moves to v2's required lookback parameter (6h..1y)
  replacing the removed granularity parameter. Timestep is now a
  lookback selector; the response's authoritative point spacing (also
  new in v2) is used for cache expiry and preserved alongside the
  window bounds. Chart durations map to the equivalent lookbacks
  (1d->24h, 1w->7d, 1m->30d, 1y->1y).
- Deadman has no v2 route (404), so it stays on v1.
- Display of over-max prices uses a long-capable RS decimal formatter
  (RuneLite's QuantityFormatter is int-only); chart scaling saturates
  at max cash, which is visually indistinguishable for such items.
- Null prices now read as "no data" instead of NPEing the panels.

Verified the live endpoints: v2 latest/timeseries shapes match the new
models; parsing tests cover over-max numbers and strings, decimals,
nulls, and the v2 timeseries metadata.
feat: add SQLite storage for trades and recipe flips
feat: wiki price API v2 with over-max-cash support
fix: derive SQLite JSON codec from injected Gson
…mpliance

fix: use RuneLite-provided SLF4J for SQLite
@AntonyGarand AntonyGarand self-assigned this Sep 28, 2026
@coderabbitai

coderabbitai Bot commented Sep 28, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@AntonyGarand
AntonyGarand force-pushed the refactor/json-only-storage branch from 048ef77 to ae3d8a2 Compare October 13, 2026 03:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants