git ssb


Dominic / scuttlebot

use secure-scuttlebutt/flume

Closed Dominic wants to merge commits into master from flume
Dominic · 11/29/2016, 10:46:34 AM

use secure-scuttlebutt/flume

this refactors sbot to use the breaking changes in %secure-scuttlebutt/flume branch. unfortunately npm can't install ssb:// git repos, so I couldn't put it the the package json. if you want to test, I recommend cloning secure-scuttlebutt and linking it into sbot.

cd node_modules/
ln  -s ../../secure-scuttlebutt # or whereever it is relative to here

Also, you'll need to remove the indexes so they get rebuilt... rm -rf ~/.ssb/db/*/*
(I should probably make flumeviews a bit smarter so they can do that automatically, or something)

%nBL9oNvczv/zSNaM1fyEbi4u7Qb88vTnUTPIqwl5ZUQ=.sha256 ev · 11/29/2016, 11:03:26 AM

@dominic Testing breaking changes now.

%6yKIVULNVTMHX3l1o4cGO/8VZEU0MEinLWaMSHLvb2Q=.sha256 ev · 11/29/2016, 11:07:20 AM

@dominic unable to build

> secure-scuttlebutt@15.5.1 test /home/ev/secure-scuttlebot
> set -e; for t in test/*.js; do node $t; done

  var createIndex = Reduce(1, sfunction (acc, data) {
SyntaxError: missing ) after argument list
    at Object.exports.runInThisContext (vm.js:78:16)
    at Module._compile (module.js:545:28)
    at Object.Module._extensions..js (module.js:582:10)
    at Module.load (module.js:490:32)
    at tryModuleLoad (module.js:449:12)
    at Function.Module._load (module.js:441:3)
    at Module.require (module.js:500:17)
    at require (internal/module.js:20:19)
    at module.exports (/home/ev/secure-scuttlebot/db.js:11:18)
    at module.exports (/home/ev/secure-scuttlebot/index.js:65:27)
%qVaUYm3/RIV00Q+ATkcP7wQ1PSj394nwvkm2NVtARl8=.sha256 Dominic · 11/29/2016, 5:11:09 PM

sorry, try the latest

%X/Ze7q3ucm03saI5HMILF1EPTIz+SRFYXcp8o4r0PrI=.sha256 ev · 11/29/2016, 5:55:10 PM

secure-scuttlebot builds, but now I'm getting cannot find module graphreduce in scuttlebot. Probably not published to npm yet.

%FFHpy69+u7LXr3Zi7SI8SE6WmWCJBj3qkJMEDYfYI+s=.sha256 Dominic · 11/30/2016, 3:50:35 AM

sorry, published

%YH/9Fp2w11HGAXcZLuRVbc0qKHkDQMHdk0XO+IEsBCo=.sha256 ev · 11/30/2016, 8:31:24 AM

@dominic Still getting

Registry returned 404 for GET on

%tTmgSYuKORZYqhFj85tY9Ig8Wp6i8CQDmavfAsmou90=.sha256 dinosaur · 12/1/2016, 9:29:51 PM

got it to run after changing:

diff --git a/package.json b/package.json
index e8454fe..28b3c6b 100644
--- a/package.json
+++ b/package.json
@@ -14,7 +14,8 @@
     "cont": "~1.0.3",
     "deep-equal": "^1.0.1",
     "explain-error": "~1.0.1",
-    "flumeview-reduce": "^0.2.1",
+    "flumeview-reduce": "^1.0.1",
+    "graphreduce": "^2.0.0",
     "has-network": "0.0.0",
     "ip": "^0.3.3",
     "level-memview": "~0.0.0",

it somewhat worked, as in i could get patchbay to load once, but then i kept getting expected previous: ... found: ... log statements and browsing to git ssb web wouldn't load.

also i wonder if it's possible to have a migration script, since the re-indexing loses all the "message received" timestamps, which really messes with patchbay and kinda loses previous information, but also definitely only a nice-to-have.

anyways, this is cool, i really like the direction of flumedb.

%eidgJ9Bi6V7OV432MPWKv6I4Vx/CW8cOsKyrNgSX+Aw=.sha256 ev · 12/21/2016, 5:14:47 PM

I decided to give this branch another shot after the call, but I'm getting this error and I'm not sure how to fix it:

          ssb.use(name, flumeview)

TypeError: ssb.use is not a function
    at EventEmitter._flumeUse (/home/ev/sbot/index.js:65:15)
    at EventEmitter.hooked [as _flumeUse] (/home/ev/sbot/node_modules/hoox/index.js:10:15)
    at Object.exports.init (/home/ev/sbot/plugins/friends.js:35:20)
    at /home/ev/sbot/node_modules/secret-stack/api.js:27:28
    at Array.forEach (native)
    at create (/home/ev/sbot/node_modules/secret-stack/api.js:26:20)
    at Object.<anonymous> (/home/ev/sbot/bin.js:57:16)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
%Z9DvPo3DfOqP5LQm77PogQWupAu1863OjhRdgkJQIPk=.sha256 yetAnotherLiteCryptix · 12/21/2016, 6:11:37 PM

@ev the way the plugins are uncommented is a bit clumsy, i think. trying to fix it to.

%sOJ+NQYJ3I3DcZW5oqR5S8vQzWlnZ9gDpNKOQbo5arA=.sha256 yetAnotherLiteCryptix · 12/21/2016, 6:15:06 PM

no.. wrong file, sorry. i'll get some sleep..

%vOIpEAnkzHrEQbgA+iua3q0I+UFQ1hmxO4nY4R38D2U=.sha256 Dominic · 12/22/2016, 6:07:35 AM

okay I have updated the flume branch to put the flume stuff beside the old database, not inside it, which means you can switch between them easily.

on this PR you need to simlink in secure-scuttlebutt on the flume branch.

Dominic updated the branch to cd253f5b · 12/28/2016, 8:53:04 AM
%BZn4A1souwQXeFrO+rL5+7FFJJd3MqywstpA/mXeYKM=.sha256 Dominic · 12/28/2016, 8:58:59 AM

Okay this version should now be runnable without obliterating your old version. instead of loading the flume stuff within the .ssb/db directory, it will put it in .ssb/flume this means it won't clobber your old indexes, so you can switch back easily. Also, if it isn't up to date, it will copy your old data into flume's log, so messages will still be in the same order.

cryptix mentioned this pull request in ## preshow [vimb]( !!! noffle wrote about it i3, tiling window managers pull requests prios ## from th
%wmUPmTbmc8A473k+DWyn6dqy7FdCt7lseWaBGxJwH6Q=.sha256 ev · 1/4/2017, 6:56:34 PM

Ok. @gb just tried this branch and got the

TypeError: ssb.use is not a function


I double checked my supposed working flume, and I discovered that my big error was never pulling flume's branch. So I was actually just running scuttlebot@9.4.2

I'm not sure how to fix this, but we both don't have working flume versions.

@dominic If you try a clean flume install, does it work?

%1gEVxSDAk424d62/SKTaGj6zLUht2JyS1fdjizlL5pg=.sha256 cryptix · 1/4/2017, 7:03:18 PM

Thank's for trying, you two! That is also the error that I hit last time I tried but was to rushed to document it properly.

I also went through not using the secure-scuttlebutt flume fork on npm but that didn't help either... :{

%bxx+ifTAd7o/+5mQPJcutQHe7cJOZv9rOEj7xU6N+oA=.sha256 gb · 1/4/2017, 7:04:51 PM

I got it working. Hmm.

%35YAOk5eNeqaDTSlPMrszWiTFWiu/amONQ+y+IfCw0c=.sha256 ev · 1/4/2017, 7:27:41 PM

@gb Nah, that was you pulling off my branch where I pushed a version that wasn't flume. Neither of us have it working, it turns out. Both getting ssb.use is not a function.

%xAH8G6JJKELysR+FeLynvGbEgHfQIvsvHz1jJH8LOps=.sha256 gb · 1/4/2017, 7:31:26 PM

Is mercury in retrograde, brah?

%9bDOjtz4416P6LPTM1JYupMdQT+muzWr5nVhKvGFHiY=.sha256 gb · 1/18/2017, 5:21:28 PM

@dominic_temp today on the call you said you'd just pushed updates to flume -- am I looking at the right branch here or did I misunderstand you on the call?

%dHrD4ADONy+bPdvCvX56Yr+QLNd9cbwItfnw9vzyzcM=.sha256 cryptix · 1/19/2017, 4:37:05 AM

Yea.. I wanted to voice this during the call yesterday (but two clients forbid me from speaking..).

I will try to document by steps more cleanly but 3 times I hit a will with this error.

%UoW6GnVw8D8k4goS3KjQkT/5gJySp53qj775k+1Nf6U=.sha256 cryptix · 1/20/2017, 3:15:20 AM

cc @dominic_temp

%eseGnFrq/Qeg73Lx/cidTI0xC0XIpf9rIaq+A9XSl+4=.sha256 cryptix · 2/1/2017, 3:06:39 PM

ping @dominic

kinda sure I had the correct branch at some point.. :-/ trying again now.

Dominic updated the branch to 48310005 · 2/2/2017, 2:36:56 AM
%t2kwrLs2Yl3LQVmV4sx5SXI2zEW427VPG/aisT1nM+s=.sha256 Dominic · 2/2/2017, 2:42:18 AM

@cryptix @ev @gb this should just be a clone and npm install now. It might take a little while as it moves everything across into the flume database though - if you want to see if something is happening, use sbot since to show where the various flume indexes are up to. If they all show the same number then the database should be good to go!

%jzfWDhlwvo27VbC+JfxWl2A08pYd5HrMJjBsupQQj5M=.sha256 ev · 2/2/2017, 4:33:37 PM

@dominic I have a working flume! I had to restart flume a bunch of times because it kept blowing the stack, but eventually everything transitioned over to the new db.

Dominic updated the branch to 0cd58652 · 4/4/2017, 1:01:55 AM
Dominic updated the branch to b3923852 · 5/21/2017, 12:53:42 PM
Dominic deleted the flume branch · 6/23/2017, 7:37:34 PM

Built with git-ssb-web