Category: Draft

Karma score (contribution analysis system)

ChangeLog:

17 October, 2013: Initial draft (Tamer)

Introduction:

Karma score is a contribution analysis scoring system that help identifying if a user is active contributor to a community by give a score to certain activities. For example when a user login to the community portal it would award them certain score for logging in (for example 20 points), when a user reviews a comment and give it a score they would be given another higher score, and the same goes to posting reviews or comments but with higher score.

This activities would then recorded in a database table (or any other persistent store so that later when a function requires the Karma score of a user it will easily find it on that table.

At anytime, Karma score would be the sum of all individual score awarded to the user by different subsystems and categories of the community portal.

How it works:

Create a database table with the following fields:

  • User ID: ID representing the user in the system
  • Date and time: Date and time user was awarded Karma
  • Action description: Type of action done by user, for example login, comment review or posting comments etc.
  • Karma award: Score awarded for this action, which is a numeric value

Assume there is a subroutine called add_karma(user_id, action_descr, karma) which takes as parameter name suggest three values to insert them in the Karma table.

So for example when at user login, the function gets called from the login code as:

add_karma(1001, “Login action”, 20);

Which adds a record in the Karma table for the user (ID 1001) awarding them a 20 points for logging in.

So at any point in time whenever a subsystem wants Karma score for a user they would call another subroutine (get_karma_score())that would sum all records in the Karma table and returns it as Karma score for the user.

Two or more time threshold should be defined by the system, one for expiring old records for all users and the other(s) define time threshold for decreasing action scores, for example, after 10 days decrease login score from 20 to 10, etc.

A nightly job should run on the table to honour those thresholds and recalculate new Karma scores if any.

Krama decays over time:

Karma score should be influenced by the time a user performed the actions they got awarded to, for example, a user posted a review in the past 10 days should have more Karma score than a user who did a post last year. A nightly job should crawler the Karma log table and expire actions falling beyond certain time threshold, it might also decrease scoring of actions for users if they passed certain time threshold as well. So than a user logged in today would have Karma score more than a user who logged in 3 weeks ago.

This would reflect if a user is active and heavily contributing to the community or just a visitor.

Usage:

Karma score is a contribution weight, which means it reflects how a user is active and contributing to the community. The higher the score the more contribution user has performed.

Users with higher Karma score is designated to be active users, which is a personal evaluation to the activities done by user, if combined with other factors like Credibility score (how other evaluate my contribution) or Spam/report counter would give more insight of how legitimate their future contribution are or if they are worthy or not.

References:

Credibility Score Algorithm

Introduction:

The idea behind this draft blog is to introduced a system to give Credibility Score (CS) to each subscriber to any social network or collaboration portal based on their contribution to the community. Their contribution here might be posting an article or a review, how the system builds CS for that user will be based upon subsequent reactions by other members and the amount of posts performed by that user.

How it works:

Whenever a user make any post of any kind in a social network, other users using this network will be presented with a rating mechanism to rate this post, the values used to rate the post are listed below and given weight for each category. By using all ratings the user has got for his post, the below process takes place to calculate CS for this post.

Credibility Score of a user is the sum of all CS for each post they earn. CS of a user can be then used to give weight for subsequent posts of the user, CS is a number between 0 and 1, where 0 means the user has no credibility and 1 means they have the maximum credibility.

Ratings:

Each rating will be allowed one of the following values:

Rating Weight
True 1.00
Mostly true 0.75
Half true 0.5
Mostly false 0.25
False 0
Ridiculous -1

Review credibility score:

  • For each review calculate total rating and divide them into 6 categories: true, mostly true, half true, mostly false, false and ridiculous
  • Calculate proportion value and credibility score for each category, for example, if a review has 29 ratings:
    Category Category weight Category count Proportion Credibility
    True 1 4 0.13 0.13
    Mostly true 0.75 16 0.55 0.41
    Half true 0.50 7 0.24 0.12
    Mostly false 0.25 2 0.06 0.01
    False 0 1 0.03 0
    Ridiculous -1 0 0 0
    0.67
  • So the user has total of 0.67 credibility for this review

User credibility score:

  • Get total number of reviews committed by a user (for example a user has 5 reviews)
  • Get the total number of rating for user reviews (for example a user got 3 out of 5 reviews rated)
  • Calculate evaluation ratio (number of reviews rated / total number of reviews), for our example it’s ⅗ = 0.60
  • Calculate average credibility score for all rated reviews, for example of those 3 rated reviews got credibility scores: 0.67, 0.25 and 0.41 which is
    • (0.67 + 0.25 + 0.41) / 3 = 0.44
    • User unnormalized credibility score is 0.44
  • Multiply evaluation ratio with unnormalized credibility score to get the final credibility score for a user, so
    • User credibility score: 0.60 * 0.44 = 0.26
  • The more ratings a user get the higher their credibility score preserved


Note: might have issues for example if a user have only 1 review rated while they posted 75 reviews, so it doesn’t really reflect their true credibility

So for evaluating ratings done for schools, credibility score for a user will affect this rating by multiplying each rating done by the user with his/her credibility score, for example if a user give the following ratings:

Category Rating (out of 5) Rating after applying CS
Cleanliness 4 1.04
Transportation 5 1.3
Teachers 3 0.78

References: