git ssb


10+

Dominic / patchbay



Use fulltext.search RPC method for search, if it is available

%uJvAji4Ye8p7mINJjBypR2lmEcrKsYqYuVRhVcJHAD8=.sha256
Closed cel wants to merge commits into master from fulltext-search
cel · 1/18/2017, 10:27:18 PM

Use fulltext.search RPC method for search, if it is available

as mentioned in %AJnQiVX....

for use with %ssb-fulltext

cc @mixmix @ev @dominic @arj

%SI/g29LpKNcy8G5W5E/dsQtk3HCqEB270axYyvXc8GQ=.sha256 arj · 1/19/2017, 12:10:32 PM

I'm trying this PR and just searched for something random using ?tor in the search box and got the following error:

Error: not ready at Object.search (/home/arj/.ssb/node_modules/ssb-fulltext/fulltext.js:78:37) at Object.search (/home/arj/.ssb/node_modules/ssb-fulltext/index.js:29:25) at Object.hooked (/home/arj/.nvm/versions/node/v7.3.0/lib/node_modules/scuttlebot/node_modules/hoox/index.js:10:15) at Object.localCall (/home/arj/.ssb/node_modules/ssb-ws/node_modules/muxrpc/local-api.js:31:31) at Object.<anonymous> (/home/arj/.ssb/node_modules/ssb-ws/node_modules/muxrpc/local-api.js:37:22)

%vEghUJgbcnyuLsgg8d/r+yItW/2hbvc3TXYehts9WDQ=.sha256 cel · 1/19/2017, 1:16:40 PM

@arj that is from the older ssb-fulltext. i've merged the latest now into master, tagged v1.0.0 and published on npmjs.com. try again with the updated one? it uses a different fulltext engine so will have to rebuild the indexes. i think queries will still work while the indexes are building though.

%A9hnRQFxZXmV7H2NSzOC9QHQrSkkysW/8veXKZwFpOU=.sha256 arj · 1/19/2017, 1:26:48 PM

Its alive! :frog: Really cool stuff. Now we just need to make aware that you can search using ? :D

%gPIM5jzQo7WUZTVuPPnD7q46C/yL3WHDjErE9SQKT3g=.sha256 arj · 1/19/2017, 1:40:03 PM

Wow, this is REALLY fast!

Also searching for #something doesn't seem to work, but maybe that's just how #works as tagging?

%lmXKqFCMpU5JH8mg47Ci24VEIFqe/MHK0/3B6aDw1NM=.sha256 cel · 1/19/2017, 2:19:56 PM

#works will let show you the #works channel. ?#something will search for #something. (try ?#music).

agree about needing to make ?... more discoverable

%krWMeq2ieqL4Liv05Wslf/sN+EpXgRitvbo4ijzOvx4=.sha256 John · 2/4/2017, 5:40:47 PM

patchwork-next implements search on anything that doesn't have a sigil (# & %) - I think it's a much kinder default and wouldn't cost us anything to implement (that I can see).

Reviewing code now while some index builds. Sidenote - when sbot starts I would love a key of what the different chars spewing by are, e.g.

x : message index
+ : search index
. : message synce

(made up, I have no idea!)

%FG1J+tq7RXSOqZle2jeafeGp44f07/UF0AMojg8QhWg=.sha256 John · 2/4/2017, 5:53:55 PM

Wow it is disturbingly fast. I thought the slow search was fun, but this is more like the offline web!

Noticings :

  • highlighting
    • search 'members' and it doesn't highlight 'members' (is this a reflection of the underlying search
    • it's highlighting the channel name on messages too?
  • README
    • can you add an install not to the sbot setup (plugins.install) and maybe note which ones are optional (I don't know)
  • flume
    • is any of this going to be effected with the introduction of flume?
  • needs master merged in. I made some simple changes to the search page, lemme know if anything needs clarifying.

@cel this is awesome

John updated the branch to 07678ab9 · 2/4/2017, 6:05:46 PM
%QlWFPOV9jfHu1l24SU5VGozOCWad+lB52vJvvWHNAf0=.sha256 John · 2/4/2017, 6:11:16 PM

I've merged master in and tested it works fine with or without the plugin.

I also noticed it doesn't seem to order by time. e.g. search dog food and I get a message from 1 year ago, then 5 months ago. then 6 months ago

Generally when I search I want to search close first, then go back. Is it possible to order the output?

%nV9aDQzEA1TtAvbAxoQ2fuZnyQu6cC5JDWtDRZRqEFg=.sha256 cel · 2/4/2017, 7:15:49 PM

@mixmix thanks for testing it.

patchwork-next implements search on anything that doesn't have a sigil (# & %) - I think it's a much kinder default and wouldn't cost us anything to implement (that I can see).

Yes, i think that would be reasonable.

Sidenote - when sbot starts I would love a key of what the different chars spewing by are, e.g.

Good idea. They are each output by a different plugin as it indexes messages.

x : ssb-query
. : ssb-links
+ : ssb-fulltext

search 'members' and it doesn't highlight 'members' (is this a reflection of the underlying search

The fulltext db does preprocessing of the search query (e.g. stemming). So it may be turning the word 'members' into something that isn't in the message (i'm not sure what in this case). Or, if there is a tag/formatting break in the text (e.g. mem*bers*), I don't remember if TextNodeSearcher will catch that for highlighting. Or there is a bug somewhere else.

it's highlighting the channel name on messages too?

I guess so.

can you add an install not to the sbot setup (plugins.install) and maybe note which ones are optional (I don't know)

What do you mean?

is any of this going to be effected with the introduction of flume?

I haven't tested it but i think it should work - except that userland plugins are removed on the flume scuttlebot branch.

I also noticed it doesn't seem to order by time. e.g. search dog food and I get a message from 1 year ago, then 5 months ago. then 6 months ago

Generally when I search I want to search close first, then go back. Is it possible to order the output?

The output is ordered primarily by the fulltext engine's score value, which is supposed to bring more relevant results to the top. However, I agree that ordering by timestamp might be better. The code that does this is at ssb-fulltext/fulltext.js:48.

%dvs0FJvJBZ7nj8UZWKIxaf1jd/+GFuz3Eg/XkJ430f8=.sha256 Dominic · 2/4/2017, 7:53:50 PM

this already uses the pattern flume is based on, so it will be easy to port to flume and get slightly simpler.

%EUChmaoke0tKYW7/lN0zhqFmmuCJZpUEhuSIxYHUFN4=.sha256 Dominic · 2/4/2017, 8:18:34 PM

I also tested it! very fast! (except for building the indexes...) one small problem: you need a signal to show when a search result hits nothing, otherwise you can't tell the difference between found nothing and slow.

%fT/dDQxiRFR7bBMgrW+LsYAkQDlKD9e2aJFhfN+iPOU=.sha256 John · 2/4/2017, 8:22:43 PM

@cel install not == install note
At the moment this would be a hidden feature because it's not mentioned in the README.md

Cool, @cel do you want to make that changes that seem most sensible / you have time for and then lets get this merged. Did you want me to do anything with that file you linked me to ? ?

%yExFcnZxkVUqNWnyr2FsjnbOtP5YknKe+KiUjtusuT8=.sha256 cel · 2/5/2017, 10:40:03 AM

@mixmix oh, you mean a note in patchbay to install ssb-fulltext? yeah, i'll add that.

Did you want me to do anything with that file you linked me to ? ?

i was just pointing that out in case you wanted to change it. i'm okay with the search results either way so will probably leave it as is for now.

you need a signal to show when a search result hits nothing, otherwise you can't tell the difference between found nothing and slow

@dominic good point. i'll try adding a "no results found" thing

cel updated the branch to f715d01c · 2/6/2017, 8:39:08 AM
%Xcpbtk5OATCoLc6z7alg/Tv9vcppLXVDNvBorYxfq8E=.sha256 cel · 2/6/2017, 8:41:57 AM

added those changes in %xqvZ8jH..., so this should be ready to go now

John deleted the fulltext-search branch · 2/6/2017, 2:08:47 PM
%SazJ5oLCFWEumekqddV8bcFVmy8Y7ZCvqy7hp/oSE2g=.sha256 John · 2/6/2017, 2:09:09 PM

merged into 6.9.0

%c6R+sGbgPT8yLRKDJ7eZDRSPYu3OXqNAIIDa5OHULFc=.sha256 John reopened this pull request · 2/6/2017, 2:09:17 PM
%eELcLz+M4S9ERXPQqrempAokBGKpce9j0zyH+BtQcM8=.sha256 John closed this pull request · 2/6/2017, 2:09:23 PM

Built with git-ssb-web