git ssb

use secure-scuttlebutt/flume

this refactors sbot to use the breaking changes in %secure-scuttlebutt/flume branch. unfortunately npm can't install ssb:// git repos, so I couldn't put it the the package json. if you want to test, I recommend cloning secure-scuttlebutt and linking it into sbot.

cd node_modules/
ln  -s ../../secure-scuttlebutt # or whereever it is relative to here

Also, you'll need to remove the indexes so they get rebuilt... rm -rf ~/.ssb/db/*/*
(I should probably make flumeviews a bit smarter so they can do that automatically, or something)

@dominic Testing breaking changes now.

@dominic unable to build

> secure-scuttlebutt@15.5.1 test /home/ev/secure-scuttlebot
> set -e; for t in test/*.js; do node $t; done

/tmp/test-ssb-feed
/home/ev/secure-scuttlebot/indexes/last.js:19
  var createIndex = Reduce(1, sfunction (acc, data) {
                                                  ^
SyntaxError: missing ) after argument list
    at Object.exports.runInThisContext (vm.js:78:16)
    at Module._compile (module.js:545:28)
    at Object.Module._extensions..js (module.js:582:10)
    at Module.load (module.js:490:32)
    at tryModuleLoad (module.js:449:12)
    at Function.Module._load (module.js:441:3)
    at Module.require (module.js:500:17)
    at require (internal/module.js:20:19)
    at module.exports (/home/ev/secure-scuttlebot/db.js:11:18)
    at module.exports (/home/ev/secure-scuttlebot/index.js:65:27)

sorry, try the latest

secure-scuttlebot builds, but now I'm getting cannot find module graphreduce in scuttlebot. Probably not published to npm yet.

sorry, published

@dominic Still getting

Registry returned 404 for GET on https://registry.npmjs.org/graphreduce

got it to run after changing:

diff --git a/package.json b/package.json
index e8454fe..28b3c6b 100644
--- a/package.json
+++ b/package.json
@@ -14,7 +14,8 @@
     "cont": "~1.0.3",
     "deep-equal": "^1.0.1",
     "explain-error": "~1.0.1",
-    "flumeview-reduce": "^0.2.1",
+    "flumeview-reduce": "^1.0.1",
+    "graphreduce": "^2.0.0",
     "has-network": "0.0.0",
     "ip": "^0.3.3",
     "level-memview": "~0.0.0",

it somewhat worked, as in i could get patchbay to load once, but then i kept getting expected previous: ... found: ... log statements and browsing to git ssb web wouldn't load.

also i wonder if it's possible to have a migration script, since the re-indexing loses all the "message received" timestamps, which really messes with patchbay and kinda loses previous information, but also definitely only a nice-to-have.

anyways, this is cool, i really like the direction of flumedb.

I decided to give this branch another shot after the call, but I'm getting this error and I'm not sure how to fix it:

/home/ev/sbot/index.js:65
          ssb.use(name, flumeview)
              ^

TypeError: ssb.use is not a function
    at EventEmitter._flumeUse (/home/ev/sbot/index.js:65:15)
    at EventEmitter.hooked [as _flumeUse] (/home/ev/sbot/node_modules/hoox/index.js:10:15)
    at Object.exports.init (/home/ev/sbot/plugins/friends.js:35:20)
    at /home/ev/sbot/node_modules/secret-stack/api.js:27:28
    at Array.forEach (native)
    at create (/home/ev/sbot/node_modules/secret-stack/api.js:26:20)
    at Object.<anonymous> (/home/ev/sbot/bin.js:57:16)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)

@ev the way the plugins are uncommented is a bit clumsy, i think. trying to fix it to.

no.. wrong file, sorry. i'll get some sleep..

okay I have updated the flume branch to put the flume stuff beside the old database, not inside it, which means you can switch between them easily.

on this PR you need to simlink in secure-scuttlebutt on the flume branch.

Okay this version should now be runnable without obliterating your old version. instead of loading the flume stuff within the .ssb/db directory, it will put it in .ssb/flume this means it won't clobber your old indexes, so you can switch back easily. Also, if it isn't up to date, it will copy your old data into flume's log, so messages will still be in the same order.

Ok. @gb just tried this branch and got the

TypeError: ssb.use is not a function

error.

I double checked my supposed working flume, and I discovered that my big error was never pulling flume's branch. So I was actually just running scuttlebot@9.4.2

I'm not sure how to fix this, but we both don't have working flume versions.

@dominic If you try a clean flume install, does it work?

Thank's for trying, you two! That is also the error that I hit last time I tried but was to rushed to document it properly.

I also went through not using the secure-scuttlebutt flume fork on npm but that didn't help either... :{

I got it working. Hmm.

@gb Nah, that was you pulling off my branch where I pushed a version that wasn't flume. Neither of us have it working, it turns out. Both getting ssb.use is not a function.

Is mercury in retrograde, brah?

@dominic_temp today on the call you said you'd just pushed updates to flume -- am I looking at the right branch here or did I misunderstand you on the call?

Yea.. I wanted to voice this during the call yesterday (but two clients forbid me from speaking..).

I will try to document by steps more cleanly but 3 times I hit a will with this error.

cc @dominic_temp

ping @dominic

kinda sure I had the correct branch at some point.. :-/ trying again now.

@cryptix @ev @gb this should just be a clone and npm install now. It might take a little while as it moves everything across into the flume database though - if you want to see if something is happening, use sbot since to show where the various flume indexes are up to. If they all show the same number then the database should be good to go!

@dominic I have a working flume! I had to restart flume a bunch of times because it kept blowing the stack, but eventually everything transitioned over to the new db.