Assigning scores about brightid-node HOT 8 CLOSED

so the first thing we have to do is to save raw ranks instead of normalized ranks. I will do that today.

secondly, if i'm not mistaken, you intend to adjust the boundries manually in each iteration by relying on the scores that known honests (seeds?) have gained.

Should these calculations be done every time the SybilGroupRank run and the normalized result be saved in db for each group? Or they are done on the API side at the time of reading in the nodejs on the backend?

from brightid-node.

They boundaries should be adjusted every time sybilgrouprank is run. They can be persisted somewhere where the node can access them (as part of serving a request for a score)--the DB makes sense.

As far the boundary-setting function, I'll create a document we can work on together. https://docs.google.com/document/d/1sGZeivcVFUiT9erYKmo07w98GZ0pkTWLWrsF5I3XJTA/edit?usp=sharing

from brightid-node.

boundaries should be adjusted every time sybilgrouprank is run.

manually? or by a script looking at seed group scores for example?

They can be persisted somewhere where the node can access them (as part of serving a request for a score)--the DB makes sense.

I can simply add a script just like label_seed_groups.py to read a csv file listing boundaries and fill a collection we define for saving boundaries.

from brightid-node.

I was reading the first comment again and I think we should clarify what we mean from the "boundaries map" term. Do we mean a map between ranges of raw ranks of each group and the probability of being honest if someone score be in that range? Or we mean a map between ranges of raw ranks and normalized ranks?

It seems you mean first one. But what we should finally achieve for distribution, save in database and use at query time to return normalized score is the second.

The goal from normalization is to convert raw scores gained from sybilrank algorithm to new normalized scores which make our honest community scores distributed like attached image.

For example if we suppose -3, 0, 3 in this image are 0, 50 and 100, we should give a normalized score to our users in a way that 68% of honest users get a score between 33 and 66.

from brightid-node.

"Normalization" was probably a poor word choice on my part.

Let me explain what we want to end up with. We want a score that clearly communicates something to application providers. What we want to communicate is this:

If you choose a score of 90 (of 100) you are saying you can tolerate 1 in 10 users being sybils. If you choose a score of 50 (of 100) you are saying you can tolerate half of your users being sybils. If you choose a score of 20, you are saying you can tolerate 4 in 5 users being a sybil.

What we really want scores to mean is "sybil tolerance." The scores are used by applications, and they are the ones that need to have an idea what they mean. To an end-user all they know is "higher is better" and it's easy to come up with a system that achieves that. To create a score that communicates sybil tolerance effectively to applications is our challenge.

from brightid-node.

"Normalization" was probably a poor word choice on my part.

my goal in implementing that normalization was to spread close users in dense areas to an extend enough to distinguish them.

The approach you are offering here seems to be very useful to act as the logic for distributing the ranks.

from brightid-node.

Right. From a perspective of utility for the app providers, it's fine if the majority of users have the same score. 80% of users could all have the highest score, and that could be fine, possibly even preferable for app providers.

The goals are laid out here (which you've seen and commented on): https://docs.google.com/document/d/1sGZeivcVFUiT9erYKmo07w98GZ0pkTWLWrsF5I3XJTA/edit?usp=sharing

I renamed the document "Assigning Scores."

from brightid-node.

Closing now that we're using "verifications."

from brightid-node.