ssb-db

secret-stack plugin which provides storing of valid ssb messages in an append-only log.

minimal ssb-db

removes everything not strictly needed for ssb-db, the goal being enough features to support replication, but compatibility with the current application stack is out of scope.

compromises for the browser

I took some shortcuts to get this working in the browser quickly.

• uses flumelog-memory (this does have persistence, but keeps everything in working memory. This is okay for small scale)
• removed all level based indexes. Currently no way to request replies or messages by type. Only message by key and messages in receive order.
• all other indexes changed to use flumeview-reduce (because then there was only one thing to polyfil)

This current version works, but isn't gonna scale well to lots of data. It would be good to use it for a new network, but harder to make it compatible with the current ssb network. (however, I believe we'll get there)

Also, some problems I noticed:

• the javascript is really big. It's mostly libsodium-wrappers though, which is 1.3 mb. I think we get that down to 200k though.

Example

In this example, we create a feed, post a signed message to it, then create a stream that reads from the feed.

/**
 * create an ssb-db instance and add a message to it.
 */

//create a secret-stack instance and add ssb-db, for persistence.
var createSbot = require('secret-stack')({})
  .use(require('ssb-db')


// create the db instance.
// Only one instance may be created at a time due to os locks on port and database files.

var sbot = createSbot(require('ssb-config'))

//your public key, the default key of this instance.

sbot.id

//or, called remotely

sbot.whoami(function (err, data) {
  console.log(data.id) //your id
})

// publish a message to default identity
//  - feed.add appends a message to your key's chain.
//  - the `type` attribute is required.

feed.publish({ type: 'post', text: 'My First Post!' }, function (err, msg, hash) {
  // the message as it appears in the database:
  console.log(msg)

  // and its hash:
  console.log(hash)
})

// stream all messages for all keypairs.
pull(
  ssb.createLogStream(),
  pull.collect(function (err, ary) {
    console.log(ary)
  })
)

Concepts

<link to scuttlebutt.nz>

Building upon ssb-db requires understanding a few concepts that it uses to ensure the unforgeability of message feeds.

Identities

An identity is simply a public/private key pair.

Even though there is no worldwide store of identities, it's infeasible for anyone to forge your identity. Identities are binary strings, so not particularly human-readable.

Feeds

A feed is an append-only sequence of messages. Each feed is associated 1:1 with an identity. The feed is identified by its public key. This works because public keys are unique.

Since feeds are append-only, replication is simple: request all messages in the feed that are newer than the latest message you know about.

Note that append-only really means append-only: you cannot delete an existing message. If you want to enable entities to be deleted or modified in your data model, that can be implemented in a layer on top of ssb-db using delta encoding.

Messages

Each message contains:

• A message object. This is the thing that the end user cares about. If there is no encryption, this is a {} object. If there is encryption, this is an encrypted string.
• A content-hash of the previous message. This prevents somebody with the private key from changing the feed history after publishing, as a newly-created message wouldn't match the "prev-hash" of later messages which were already replicated.
• The signing public key.
• A signature. This prevents malicious parties from writing fake messages to a stream.
• A sequence number. This prevents a malicious party from making a copy of the feed that omits or reorders messages.

Since each message contains a reference to the previous message, a feed must be replicated in order, starting with the first message. This is the only way that the feed can be verified. A feed can be viewed in any order after it's been replicated.

Object ids

The text inside a message can refer to three types of ssb-db entities: messages, feeds, and blobs (i.e. attachments). Messages and blobs are referred to by their hashes, but a feed is referred to by its signing public key. Thus, a message within a feed can refer to another feed, or to a particular point within a feed.

Object ids begin with a sigil @ % and & for a feedId, msgId
and blobId respectively.

Note that ssb-db does not include facilities for retrieving a blob given the hash.

Replication

<link to ssb-replicate and ssb-ebt>

It is possible to easily replicate data between two instances of ssb-db. First, they exchange maps of their newest data. Then, each one downloads all data newer than its newest data.

Scuttlebot is a tool that makes it easy to replicate multiple instances of ssb-db using a decentralized network.

Security properties

ssb-db maintains useful security properties even when it is connected to a malicious ssb-db database. This makes it ideal as a store for peer-to-peer applications.

Imagine that we want to read from a feed for which we know the identity, but we're connected to a malicious ssb-db instance. As long as the malicious database does not have the private key:

• The malicious database cannot create a new feed with the same identifier
• The malicious database cannot write new fake messages to the feed
• The malicious database cannot reorder the messages in the feed
• The malicious database cannot send us a new copy of the feed that omits messages from the middle
• The malicious database can refuse to send us the feed, or only send us the first N messages in the feed
• Messages may optionally be encrypted. See test/end-to-end.js.

API

SecretStack.use(require('ssb-db')) => sbot

Adds ssb-db persistence to a secret-stack setup. Without other plugins, this instance will not have replication or querying. Loading ssb-db directly is useful for testing, but it's recommended to instead start from a plugin bundle like ssb-server

Because of legacy reasons, all the ssb-db methods are mounted on the top level object, so it's sbot.get instead of sbot.db.get as it would be with all the other ssb-* plugins.

sbot.get (id | seq | opts, cb)

Get an ssb message. If id is a message id, the message is returned. If seq is provided, the message at that offset in the underlying flumelog is returned. If opts is passed, the message id is taken from either opts.id or opts.key. If opts.private = true the message will be decrypted if possible. If opts.meta = true is set, or seq is used, the message will be in {key, value: msg, timestamp} format. Otherwise the raw message (without key and timestamp) are returned. This is for backwards compatibility reasons. Given that most other apis (such as createLogStream) by default return {key, value, timestamp} it's recommended to use get({id: key, meta: true}, cb)

sbot.add(msg, cb)

append a raw message to the local log. msg must be a valid, signed message. ssb-validate is used internally to validate messages.

sbot.publish(content, cb)

create a valid message with content with the default identity and append it to the local log. ssb-validate is used to construct a valid message.

sbot.whoami(cb)

call back with the default identity for the sbot.

ssbDb#createLogStream({lt,lte,gt,gte: timestamp, reverse,old,live,raw: boolean, limit: number, private: boolean}) => PullSource

sbot.createRawLogStream (lt,lte,gt,gte: offset, reverse,old,live: boolean, limit: number, private: boolean})

provides access to the raw flumedb log. ranges refer to offsets in the log file.

messages are returned in the form:

{
  seq: offset,
  value: {key: Hash, value: Message, timestamp: timestamp}
}

all options supported by flumelog-offset are supported.

ssbDb#addMap (fn)

Add a map function to be applied to all messages on read. The fn function is should expect (msg, cb), and must eventually call cb(err, msg) to finish.

These modifications only change the value being read, but the underlying data is never modified. If multiple map functions are added, they are called serially and the msg output by one map function is passed as the input msg to the next.

Additional properties may only be added to msg.value.meta, and modifications may only be made after the original value is saved in msg.value.meta.original.

ssbDb.addMap(function (msg, cb) {
  if (!msg.value.meta) {
    msg.value.meta = {}
  }

  if (msg.value.timestamp % 3 === 0)
    msg.value.meta.fizz = true
  if (msg.timestamp % 5 === 0)
    msg.value.meta.buzz = true
  cb(null, msg)
})

const metaBackup = require('ssb-db/util').metaBackup

ssbDb.addMap(function (msg, cb) {
  // This could instead go in the first map function, but it's added as a second
  // function for demonstration purposes to show that `msg` is passed serially.
  if (msg.value.meta.fizz && msg.value.meta.buzz) {
    msg.meta = metaBackup(msg.value, 'content')

    msg.value.content = {
      type: 'post',
      text: 'fizzBuzz!'
    }
  }
  cb(null, msg)
})

_flumeUse(name, flumeview) => view

Add a flumeview to the current instance. This method was intended to be a temporary solution, but is now used by many plugins, which is why it starts with _.

see undocumented creating a secret-stack plugin.

getAtSequence ([id, seq], cb(err, msg))

get a message at a given feed id with given sequence. calls back a message or an error, takes a two element array with a feed id as the first element, and sequence as second element.

needed for ssb-ebt replication

getVectorClock (cb)

load a map id to latest sequence ({<id>: <seq>,...}) for every feed in the database.

needed for ssb-ebt replication

progress

return the current status of various parts of the scuttlebut system that indicate progress. This api is hooked by a number of plugins, but ssb-db adds an indexes section. (which represents how fully built the indexes are)

the output might look like:

{
  "indexes": {
    "start": 607551054,
    "current": 607551054,
    "target": 607551054
  },
}

progress is represented linearly from start to target. Once current is equal to target
the progress is complete. start shows how far it's come. The numbers could be anything, but start <= current <= target if all three numbers are equal that should be considered 100%

status

returns metadata about the status of various ssb plugins. ssb-db adds an sync section, that shows where each index is up to. output might took like this:

{
  "sync": {
    "since": 607560288,
    "plugins": {
      "last": 607560288,
      "keys": 607560288,
      "clock": 607560288,
      "time": 607560288,
      "feed": 607560288,
      "contacts2": 607560288,
      "query": 607560288,
      ...
    },
    "sync": true
  }
}

sync.since is where the main log is up to, since.plugins.<name> is where each plugin's indexes are up to.

version

return the version of ssb-db. currently, this returns only the ssb-db version and not the ssb-server version, or the version of any other plugins. We should fix this soon

queue (msg, cb)

add a message to be validated and written, but don't worry about actually writing it. the callback is called when the database is ready for more writes to be queued. usually that means it's called back immediately.

not exposed over rpc.

flush (cb)

callback when all queued writes are actually definitely written to the disk.

Obv: post (fn({key, value: msg, timestamp}))

observable that calls fn
whenever a message is appended (with that message)

not exposed over rpc.

Obv: since (fn(seq))

an observable of the current log sequence. This is always a positive integer that usually increases, except in the exceptional circumstance that the log is deleted or corrupted.

addUnboxer ({key:unboxKey, value: unboxValue})

add an unboxer object, any encrypted message is passed to the unboxer object to test if it can be unboxed (decrypted)

unbox (data, key)

attempt to decrypt data using key. Key is a symmetric key, that is passed to the unboxer objects.

depricated apis

Stability

Stable: Expect patches, possible features additions.

License

MIT