[shrayarora8]

Depends on apache/incubator-resilientdb#269. The first four commits belong to that PR.

What this PR does

#269 added four commands to the
server: CREATE_INDEX_ENTRY, DELETE_INDEX_ENTRY, UPDATE_INDEX_ENTRY and QUERY_BY_INDEX.
This PR adds the three ways an application can call them through the C++ client
(KVClient), the command line tool, and the Python module. It also adds an end-to-end test and a
benchmark against a running cluster.

Today, to answer "which users live in Davis?",an app fetches every record and filters them in its own code, or copies everything into MongoDB and queries it there. After this PR the app can ask ResilientDB directly, in one call.

Which apps can use it after this PR, and which still can't. Anything that links to the C++ client
or uses the Python module can use it now. Apps that talk to ResilientDB over HTTP (ResLens, resdb-orm, the Rust SDK and ResVault) cannot yet. These apps go throughecosystem/graphql/service/http_server/crow_service.cpp, which uses its own client, sdk::ResDBKVClient, and that class does not inherit from KVClient. Giving it the four methods and adding one route is the next PR.

Using it

Command line. rdb below is kv_service_tools --config <client config>:

$ rdb --cmd set --key user:1 --value '{"name":"Ana","city":"Davis"}'
$ rdb --cmd set --key user:2 --value '{"name":"Bo","city":"Davis"}'

$ rdb --cmd create_index_entry --index by_city --attrs Davis --pk user:1
create_index_entry index = by_city, attrs = Davis, pk = user:1, ret = 0

$ rdb --cmd query_by_index --index by_city --attrs Davis
query_by_index index = by_city, attrs = Davis, 2 result(s)
user:1
user:2

$ rdb --cmd query_by_index --index by_city --attrs Davis --with_values
query_by_index index = by_city, attrs = Davis, 2 result(s)
user:1 = {"name":"Ana","city":"Davis"}
user:2 = {"name":"Bo","city":"Davis"}

Python.

import kv_operation as kv

kv.set_value("user:1", '{"name":"Ana","city":"Davis"}', config)
kv.create_index_entry("by_city", "Davis", "user:1", config)

kv.query_by_index("by_city", "Davis", config)
# ['user:1', 'user:2']

kv.query_records_by_index("by_city", "Davis", config)
# [('user:1', '{"name":"Ana","city":"Davis"}'), ('user:2', '{"name":"Bo","city":"Davis"}')]

One attribute can be a plain string, as above. For an index with several attributes, you can pass a list like,
kv.query_by_index("by_owner_format", ["shray", "jpeg"], config).

C++.

KVClient client(config);
client.CreateIndexEntry("by_city", {"Davis"}, "user:1");

std::unique_ptr<Items> matches = client.QueryByIndex("by_city", {"Davis"});
// matches->item(0).key() == "user:1"

An index with more than one attribute

Say a photo app stores photos and wants to find them by owner and by file format. It creates an
index called by_owner_format and gives each photo two attributes, owner first, format second:

create_index_entry by_owner_format  shray,jpeg  photo:1
create_index_entry by_owner_format  shray,png   photo:2
create_index_entry by_owner_format  shray,jpeg  photo:3
create_index_entry by_owner_format  ana,jpeg    photo:4

Now:

$ rdb --cmd query_by_index --index by_owner_format --attrs shray
photo:1  photo:2  photo:3          # everything shray owns

$ rdb --cmd query_by_index --index by_owner_format --attrs shray,jpeg
photo:1  photo:3                   # shray's jpegs

$ rdb --cmd query_by_index --index by_owner_format --attrs jpeg
                                   # nothing

Entries are stored asindex, then the attributes in the order you gave them, then the primary key, and the server
answers a query by seeking to the entries that start the same way. Owner comes first, so all of
shray's photos sit together and are cheap to find. Jpegs are spread across every owner, so there
is no single run of them to jump to. An app that needs "all jpegs regardless of owner" creates a
second index with format first.

Design decisions

1. The three new index commands return a new code (-3) when the server refuses them. KVClient's
   existing methods either return data or cannot fail, so there was no way to say "the servers
   said no". CreateIndexEntry, DeleteIndexEntry and UpdateIndexEntry return 0 on success
   and -3 when the servers refused the request, for example indexing a record that does not
   exist. Anything else is passed through unchanged from
   TransactionConstructor::SendRequest: -1 if the request could not be sent or no reply came
   back, -2 if the reply could not be parsed. -3 is the only new value, and nothing else in
   interface/, api/ or service/tools/kv/ uses it, so no existing caller changes is affected.

   On the command line that looks like this (user:999 was never stored):

   $ rdb --cmd create_index_entry --index by_city --attrs Davis --pk user:999
   create_index_entry index = by_city, attrs = Davis, pk = user:999, 
   ret = -3 (rejected by the servers)
   

   If you would rather not add a return code, two alternatives are small changes and I am happy to
   switch:

   • have the server send back an error string, which also says why it refused
   • every failure to -1, which keeps the existing vocabulary but tells the caller less.

2. These commands wait for the reply while set does not. set returns as soon as the request
   leaves the client, which I confirmed by killing two of four replicas: it still returned 0
   although nothing could commit. That is fine for set, which has nothing to report. Since in this design, the server can refuse these commands, so the four index methods wait for the answer just like howget works today.

Benchmark

//benchmark/e2e:composite_key_e2e_benchmark compares the old way of answering a question with
the new one, against a running cluster.

The setup. Four replicas and one client proxy on a single machine, storing to LevelDB. The
benchmark writes N records, each a JSON document of a chosen size, and gives 1% of them
city = Davis. Then it times two ways of answering "which records have city = Davis?", runs
each three times and keeps the fastest:

• fetch everything and filter, which is what an application has to do today,
• one query_by_index call, which is what this PR adds.

Records and size	Data in the store	Fetch everything and filter	Query the index, keys only	Query the index, with values
5,000 × 64 B	0.3 MB	203 ms	176 ms (1.2×)	166 ms (1.2×)
5,000 × 2 KB	10 MB	303 ms	174 ms (1.7×)	159 ms (1.9×)
5,000 × 16 KB	80 MB	1,302 ms	140 ms (9.3×)	218 ms (6.0×)
10,000 × 16 KB	160 MB	2,420 ms	147 ms (16.5×)	207 ms (11.7×)

Why secondary index improves with document size. Fetching everything has to carry every record across the
network, so its cost rises with the amount of data in the store. Here, we see 203 ms at 0.3 MB and 2,420 ms at
160 MB. The index query only touches the entries that match, 50 or 100 of them, so its cost barely
moves: 176 ms down to 147 ms across the same range. At small record sizes there is almost nothing
to win, because every request pays a fixed cost of roughly 150 ms (the proxy waits for 100 requests
or 100 ms before starting consensus, and a lone client pays that wait), and 0.3 MB of data is
cheap to move next to it. At 16 KB documents, the store holds 160 MB and moving all of it is the
whole cost.

On fetching the matching records. A query returns primary keys. If the app then wants the
records themselves, it does a get per key, and each one is a separate request through consensus,
about 200 ms here. For two or three matches that is fine. For 100 matches it is about 18 seconds,
which is slower than fetching the whole database in one request. That is the reason I added
with_values in #269. Using withvalues, the server reads the matching records itself and returns them in the same reply, so the app pays one request no matter how many records match.

The trade-off. with_values sends back every matching record, so a query that matches most of
the store returns most of the store. With half the records matching I measured 316 ms for fetching
everything against 228 ms with with_values, so it still wins, but the margin narrows. Refer to the
open questions.

Tests

service/tools/kv/api_tools/composite_key_e2e_test.sh starts a cluster with LevelDB, runs 20
checks against it, and prints PASS or the first check that failed. It exits non-zero on failure
so CI can run it. What it covers:

• Records go in and come back. Three records are stored and each one read back.
• Queries return the right rows. By one attribute, by two attributes, the whole index with no
  attributes, and a query that must return nothing.
• The three ways a command can be refused, each checked to return ret = -3 and to leave the
  index exactly as it was:
  • creating an entry for photo:999, a record that was never stored,
  • deleting an entry that was never created,
  • updating from attributes that match no existing entry.
• Update and delete. An update moves an entry from one attribute to another. A delete removes
  the entry but leaves the record: get still returns it afterwards.
• Index entries stay out of ordinary reads. A range scan over every key returns only user
  records.
• The index is on disk. The cluster is restarted and the same query still answers correctly.

Every write is confirmed by reading it back, because set does not wait for consensus. I also
checked the script fails when it should, by feeding it a wrong expected value and watching it
report the failure.

Note on existing behaviour

Nothing that exists changes. The CLI gains commands and flags and its existing flags and the older
positional form are untouched. The Python module keeps get and set and adds four functions.
KVClient's existing methods are unchanged.

Future work

1. The HTTP layer, so ResLens, resdb-orm, ResVault and the Rust SDK can use this. It needs the
   four methods on sdk::ResDBKVClient and one route in crow_service.cpp. Small, and it is what
   makes the feature reachable for the apps Proposal to Add Interface to Create and Retrieve CompositeKeys #180 names.
2. Automatic index maintenance. If an app updates a record and forgets to update its index
   entries, the index keeps pointing at the old attributes and queries return stale results, and
   nothing detects it. One possible fix is an opt-in command that writes the record and fixes its
   index entries in a single atomic write, so the two cannot drift apart, and which costs one
   request instead of two. I will write this up as a separate proposal.
3. Benchmark across separate machines, where each request costs a real network round trip
   rather than a loopback one.

Open questions

1. Should with_values cap what it returns? Today a query matching most of the store returns
   most of the store in one reply. A limit on the number of records, or on total size, with a flag
   saying the result was truncated, would make the worst case predictable. I did not add one
   because I did not want to invent a policy on my own.
2. Is -3 the right way to report a refusal?
3. Should the CLI's --attrs support an escape? It splits on commas, so an attribute value
   containing a comma cannot be expressed today.

Files touched

New: service/tools/kv/api_tools/composite_key_e2e_test.sh,
benchmark/e2e/composite_key_e2e_benchmark.cpp and its BUILD.

Modified: interface/kv/kv_client.{h,cpp}, service/tools/kv/api_tools/kv_service_tools.cpp,
api/pybind_kv_service.cpp, api/kv_operation.py.

Refs #180, builds on #269