Skip to content

gregdaynes/hold-this

Repository files navigation

Hold This

A simple key-value store that uses SQLite as the backend. It is designed to be used in a single-threaded synchronous environment.

Getting Started Examples

Getting Started

  1. Add dependency to project

     npm install --save hold-this
    
  2. Create instance of hold-this

     import Hold from 'hold-this'
    
     const holder = Hold()
    
  3. Write data to store

     holder.set('accounts', 'account-123:user-123:name', 'Alice')
    
  4. Read data from store

     holder.get('accounts', 'account-123:*:name')
    

Examples

No Config Setup

import hold from 'hold-this'

const holder = hold()
holder.set('accounts', 'account-123:user-123:name', 'Alice')
holder.set('accounts', 'account-123:user-456:name', 'Bob')

console.log(holder.get('accounts', 'account-123:*:name'))
// => [['account-123:user-123:name', 'Alice'], ['account-123:user-456:name', 'Bob']]

File backed store

Pass an object with a key location and a path to a file. This will be the filepath that hold-this utilizes to write to disk.

import hold from 'hold-this'

const holder = holder({ location: './holder.sqlite' })
holder.set('accounts', 'account-123:user-123:name', 'Alice')
holder.set('accounts', 'account-123:user-456:name', 'Bob')

console.log(holder.get('accounts', 'account-123:*:name'))
// => [['account-123:user-123:name', 'Alice'], ['account-123:user-456:name', 'Bob']]

File based store defaults to using WAL for performance purposes. This can be disabled by setting { enableWAL: false} when creating the instance.

Caution

This will severely decrease write performance for File based storage.

import hold from 'hold-this'

const holder = holder({ location: './holder.sqlite', enableWAL: false })

Tip

You can benchmark by running npm run test:bench

Task Name ops/sec Average Time (ns) Margin Samples
SET: disk 4,479 223243.780 ±3.51% 2241
SET: diskWAL 46,920 21312.531 ±1.13% 23461
SET: memory 97,835 10221.214 ±2.13% 48918
SET: memory:serialization:json 88,712 11272.352 ±2.78% 44357
SET: memory:serialization:json:fast 88,407 11311.238 ±3.97% 44204
SET: memory:complex-key 72,309 13829.498 ±4.37% 36155
SET: memory:buffered 83,829 11928.928 ±6.39% 41915
SET: memory:buffered + turbo 189,670 5272.298 ±4.92% 94836
GET: disk 51,374 19464.797 ±5.57% 25688
GET: diskWAL 86,877 11510.509 ±5.00% 43439
GET: memory 87,775 11392.719 ±7.03% 43888
GET: memory:serialization:json 82,347 12143.596 ±6.20% 41174
GET: memory:serialization:json:fast 81,565 12260.154 ±6.61% 40972
GET: memory:complex-key 67,595 14793.962 ±6.60% 33798

Performed on Macbook Pro M1 with 16 GB Memory

HTTP Server

Because Hold This is based on SQLite3, it does not support a native network connection. We can achieve a networked, multi-connection instance by wrapping Hold This in a HTTP Server.

Caution

This is not a production ready example. Security, and failure modes must be considered, but are outside the scope of the example.

import { createServer } from 'node:http'
import Hold from 'hold-this'

const server = createServer()
server.holder = Hold()

server.on('request', (req, res) => {
	let body = ''
	req.on('data', (chunk) => { body += chunk })

	req.on('end', () => {
		const parsedBody = JSON.parse(body)

		const { cmd, topic, key, value, options } = parsedBody
		const data = server.holder[cmd](topic, key, value, options)

		res.writeHead(200, { 'Content-Type': 'application/json' })
		res.end(JSON.stringify(data))
	})
})

server.listen(3000)

The complete example including client and benchmarks can be found in examples/http

WebSocket Server

Because Hold This is based on SQLite3, it does not support a native network connection. However we can achieve a networked, multi-connection instance by wrapping Hold This in a WebSocket Server.

Caution

This is not a production ready example. Security, and failure modes must be considered, but are outside the scope of the example.

/* global WebSocket */
import { WebSocketServer } from 'ws'
import Hold from 'hold-this'

const server = new WebSocketServer(3000)
server.holder = Hold()

server.on('connection', function connection (ws) {
	ws.on('error', console.error)

	ws.on('message', function message (payload) {
		const message = JSON.parse(payload)

		const { id, cmd } = message
		const { topic, key, value, options } = message.data

		const data = server.holder[cmd](topic, key, value, options)

		ws.send(JSON.stringify({ id, cmd, data }))
	})

	ws.send('connected')
})

export default server

The complete example including client and benchmarks can be found in examples/websocket

Bind Topic / Shorthand

Calling .bind('myTopic') on your hold-this instance, will return a modified instance that has topic already defined on set/get methods.

import hold from 'hold-this'

const holder = holder().bind('accounts')
holder.set('account-123:user-123:name', 'Alice')
holder.get('account-123:user-456:name', 'Bob')

console.log(holder.get('account-123:*:name'))
// => [['account-123:user-123:name', 'Alice'], ['account-123:user-456:name', 'Bob']]

Data Serialization

When passing the value with .set, if the value is not a string, the data will be serialized with serialize-javascript and then stored. Passing an options object like { isJSON: true }, with a proper JSON object, will signal to the serializer to use a faster mechanism.

import hold from 'hold-this'

const holder = holder().bind('accounts')
holder.set('account-123:user-123:name', { firstName: 'Alice' }, { isJSON: true })
holder.get('account-123:user-456:name', 'Bob')

console.log(holder.get('account-123:*:name'))
// => [['account-123:user-123:name', { firstName: 'Alice' }], ['account-123:user-456:name', 'Bob']]

TTL / Expiring Records

When setting a record, specifying in a options object { ttl: 1000 } will set a date in the future where the record will no longer be retrievable. Note: TTL value is set in milliseconds.

import hold from 'hold-this'

const holder = hold()
holder.set('accounts', 'account-123:user-123:name', 'Alice', { ttl: 1000 })
holder.set('accounts', 'account-123:user-456:name', 'Bob', { ttl: 0 })

console.log(holder.get('accounts', 'account-123:*:name'))
// => [['account-123:user-123:name', 'Alice']]

After a period of time, it is recommended to clean out expired records from the store. This can be achieved by calling .clean() on the instance which will remove all expired records from all topics. If a topic parameter is provided .clean('myTopic'), only this topic's expired records will be removed.

import hold from 'hold-this'

const holder = hold()
holder.set('accounts', 'account-123:user-123:name', 'Alice', { ttl: 1000 })

holder.clean()

Turbo Mode

If speed of insertion is a priority, turbo mode can be enabled. When turbo mode is used, the table is not indexed, and unique keys are disabled.

Tip

It is recommended to run turbo mode on an instance of hold-this away from the rest of your data.

Warning

Turbo mode removes table constraints and indexes. Fetching data with turbo mode enabled will be much slower.

import hold from 'hold-this'

const holder = hold({ turbo: true })
holder.set('accounts', 'account-123:user-123:name', 'Alice')

console.log(holder.get('accounts', 'account-123:*:name'))
// => [['account-123:user-123:name', 'Alice']]

Bulk Insertion

Bulk insertion leverages transactions to insert a batch of records, prepared ahead of time.

Caution

When using transactions, if one insert fails, the batch is discarded.

import hold from 'hold-this'

const holder = hold({ turbo: true })

const entries = Array.from(Array(3))
	.map((_, i) => holder.prepare('bulk', `key:${i}`, `value${i}`))

holder.setBulk('bulk', 'key', entries)

console.log(holder.get('bulk', 'key:*'))
// => [['key:0', 'value0'], ['key:1', 'value1'], ['key:2', 'value2']]
Run Time (ms) Record Count Records (per ms)
9 309.484 402861 1301.718
3 290.291 377501 1300.425
5 295.262 377501 1278.528

Using Turbo Mode Performed on Macbook Pro M1 with 16 GB Memory

Buffered Insertion

Like bulk insertion, buffered insertion uses transactions, but handles everything for you.

Everything is:

  • preparation of the insert statement. Like set
  • draining at threshold
  • draining after time threshold

Caution

When using transactions, if one insert fails, the batch is discarded.

[!INFO] Records in the buffer are not queryable.

import hold from 'hold-this'

const holder = hold({ turbo: true, bufferThreshold: 1000, bufferTimeout: 500 })

for (let i = 0; i < 1000; i++) {
	holder.setBuffered('buffer', 'key', `value${i}`)
}

// If 1000 records are not set in the buffer within the timeout of 500 ms, the buffer is drained.

console.log(holder.get('buffer', 'key:*'))
// => [['key:0', 'value0'], ['key:1', 'value1'], ...]

About

An SQLite backed KV store

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published