12

In a recent meta post, which has since changed dramatically so many comments are no longer on topic there, a moderator said a few things that I feel need to be discussed.

The following quotes are from revision 2, of the above linked post.

We've created a tool that gives us statistics on what reviews are being done, compared to what the results are. From the numbers that gives us, we calculate some more stats. Remember flag weight? Sort of like that.

Flag weight was removed over three years ago. It was removed, and there was rejoicing across the network. It was cited for many issues including:

  • Wasting moderator time
  • Flagging simply to increase flag weight
  • Actively harming the site by taking the above bullet to the extreme

Everyone has a review weight, and there's a threshold at which we'll issue a review ban. The good news is it's hard to reach that threshold. If we're handing out bans, there's something seriously wrong going on. The other good news is nobody's hit it yet.

How is this review weight calculated and how are you preventing the issues mentioned above from becoming a problem here? How is the threshold determined and how will you handle it as the community grows? Especially considering this:

The other important point to note here is we're not robots. We won't just go handing out bans for the sake of it: if someone's hit the threshold, we'll take a look and investigate first. If there are legitimate reasons why it's happened, that's fine.

It sounds like you are adding a lot more work for yourself (bullet one from above?).

My questions about this entire process, since it seems to have been announced and then removed entirely from the post making the announcement:

  • Why is the moderation team on Open Source bringing back a concept conceptually similar to flag weight - which was removed years ago?
  • Are there other communities on the Stack Exchange network utilizing this process? If so, how has it worked out for them? I think experience and information from how it's working elsewhere would be helpful to see. If not, why is Open Source going with a new process that is different from the rest of the network?
  • What is your process for determining a user's "review weight"? How flag weight was calculated was well documented. This new "review weight" needs to be documented as well. Not just to "some extent", but documented to the point where the community understands what is happening, how it is happening and why it is happening.
  • When was this going to be re-announced? It was edited into a question. It generated comments, garnered down votes and was then edited out. It does a disservice to the community for the moderators to built a way to judge the users in the community and not explain what they are doing, how they are doing it and put the only announcement that this even exists in a post that originally read as a non-specific threat to all users to be on better behavior.
5
  • 4
    I would like to add the question of what exactly was decided (as usual, preferably with a transcript, so that things don't get misunderstood or lost in translation, but I guess the answer to that will again be it was accidentally done in secret channels not open to the public)
    – Martijn
    Aug 31, 2015 at 8:39
  • 1
    @Martijn What exactly was decided w.r.t. what?
    – ArtOfCode
    Aug 31, 2015 at 10:25
  • 2
    @ArtOfCode what exactly was decided with regard to handing out review bans and/or other bans
    – Martijn
    Aug 31, 2015 at 10:28
  • 4
    You forgot "Wasting everyone's time with every declined flag causing a meta storm" for reasons to rejoice about the removal of flag weight.
    – Grace Note StaffMod
    Aug 31, 2015 at 16:42
  • 1
    We're retiring the system. What are we doing to review reviewers? Nothing more, nothing less than anywhere else on the network.
    – ArtOfCode
    Aug 31, 2015 at 17:11

1 Answer 1

Reset to default
3

IMPORTANT NOTE
This system, after the storm it's caused, has been retired. The answer below details how it worked; we're no longer using it.

First off: sorry, our fault. A lack of communication both in the team and with the community has lead to this, and we missed out on getting this information out properly.

  • Why is the moderation team on Open Source bringing back a concept conceptually similar to flag weight - which was removed years ago?

Because it's only conceptually similar. It doesn't have the problems that flag weight had, which you listed:

  • Wasting moderator time
    It's a little bit more work for us. Not massive amounts. It's work that is being done for the benefit of the site, and that's a good thing - and it's not like we have lots of other work to do. Our queues are empty!
  • Flagging simply to increase flag weight
    Is this a problem? The only way to increase your review weight is to do good reviews. If people do lots of good reviews, are we going to object?
  • Actively harming the site by taking the above bullet to the extreme
    See above. There are system limits on the review queues, which are pretty reasonable - if someone's regularly hitting them, it's not a problem.
  • Are there other communities on the Stack Exchange network utilizing this process? If so, how has it worked out for them? I think experience and information from how it's working elsewhere would be helpful to see. If not, why is Open Source going with a new process that is different from the rest of the network?

No, there aren't. It's a custom-rolled system. As to why we're using it - it brings together information from a number of sources to one place and makes it easy for us to analyse what reviews are going on, and whether anything needs to be done. We didn't know of and couldn't find any existing systems that did that, so we made our own.

  • What is your process for determining a user's "review weight"? How flag weight was calculated was well documented. This new "review weight" needs to be documented as well. Not just to "some extent", but documented to the point where the community understands what is happening, how it is happening and why it is happening.

We bring together some moderator data about the review queues, and use the last 15 reviews done by a user to figure out how many "good" vs "bad" reviews they're doing. This is done by comparing how the user reviewed it with the outcome of the review, and what state the post is currently in - it's objective, not subjective to the mods' points of view. "Good" and "bad" aren't absolutes - a bad review doesn't mean it's evil, nor does it mean the user is a complete idiot. We combine that with the number of reviews they've done, and get a review weight number. The formula looks like this:

((GOOD - BAD) / 15) * TOTAL

i.e.: a fraction representing the proportion of good reviews, multiplied by the total reviews that user has done.

With regards to what we've decided in terms of handing out review bans: we'll handle this on a case-by-case basis. Sorry, I can't produce transcripts: yes, it was in our private mod room. However, there's not much to get lost in translation. If someone is approaching 0 review weight (meaning there are about as many bad reviews as good reviews), we'll take a look into their reviewing history. Depending on what we see there, and taking into account any mitigating circumstances, we'll decide what action to take.

It's very important to note that this new metric isn't the be all and end all in handing out review bans. We're not robots; a metric can support our decisions but it'll never be the entire reason.

  • When was this going to be re-announced? It was edited into a question. It generated comments, garnered down votes and was then edited out. It does a disservice to the community for the moderators to built a way to judge the users in the community and not explain what they are doing, how they are doing it and put the only announcement that this even exists in a post that originally read as a non-specific threat to all users to be on better behavior.

That post... was a little bit of a disaster zone. The edit I made to it was hastily done in an attempt to clarify what was meant, so wasn't exactly the most polished bit of writing I've ever done.

We were going to announce this. When was as yet undefined, but "soon" would be a good approximation.

With those questions answered, I'd like to say one more thing:

This system doesn't have to be final.

If you think it's not optimal, please do tell us. Constructive criticism here can only be a good thing; we want a system that both gives us detail on what's going on (which our current design does) and that the community is happy about (i.e. not biased, etc.)

14
  • 1
    "Approaching zero"...that seems subjective. Can you elaborate on when "approaching zero" will be actioned vs something more concrete like "is zero" or "is negative"
    – Andy
    Aug 31, 2015 at 11:35
  • 1
    How often are you checking these weights? Daily? Weekly? When you get bored? Every other Tuesday if you had cake the day before?
    – Andy
    Aug 31, 2015 at 11:36
  • 1
    @Andy <= 0.5. I'd be happy to include a more concrete number, if you've got a suggestion.
    – ArtOfCode
    Aug 31, 2015 at 11:36
  • 1
    @Andy Definitely the latter. In all seriousness, I check these every couple days, and I recompile the stats each week.
    – ArtOfCode
    Aug 31, 2015 at 11:37
  • 5
    Another question that - at least to me - remains open is which problem this is trying to solve. This might be a solution in search of a problem. Our primary "problem" to me is lack of traffic and new questions. This seems to be somewhat along the lines of "We must do something. This is something. Thus we should do this".
    – Martijn
    Aug 31, 2015 at 11:52
  • 2
    "If people do lots of good reviews, are we going to object?" - Based on how this reads, yes. "This is done by comparing how the user reviewed it with the outcome of the review, and what state the post is currently in" Your good reviews can be overwhelmed by robo-reviewers. The larger sites on the network already seem a lot of 'questionable' reviews get through. Now you are potentially punishing people that review correctly but still lose the to the "community" on reviews [1/4]
    – Andy
    Aug 31, 2015 at 13:39
  • 2
    The robo reviewers, on the other hand, have all just had a successful flag. Flag weight increased! Rinse and repeat. As more and more of their reviews get accepted by other robo-reviewers that likelihood of their weight ever reaching 0 or 0.5 or whatever threshold is set gets lower and lower. You are incentivizing a problem that already exists elsewhere. Not only that, but you are allowing this bad behavior to be rewarded by explicitly saying you won't punish them because their review weight is good. [2/4]
    – Andy
    Aug 31, 2015 at 13:39
  • 1
    Take this to the next step. How do new users gain rep? Edits. Robo goes and makes an edit. It's questionable. It goes to the review and other robot-reviewers approve it. +2 for the suggestion! Rinse and repeat. Since the community agrees these are good (it passed the review process) you are saying you're not acting on it. Let this repeat for a handful of reviewers and now they can all get the association bonus. Thank you, from a moderator, for allowing users to enter my site with a head start on permissions. You could have stopped bad reviews but this new policy is saying you won't. [3/4]
    – Andy
    Aug 31, 2015 at 13:40
  • 2
    "But we review them on a case by case basis!" you say. Yes. You do. You add work to your plate. Work that could be spent elsewhere. You are giving yourself work unnecessarily. If there is a problem with reviewers (robo or otherwise) deal with them. Don't create a new threshold they have to pass. This process is going to make the community unhappy (if the communication doesn't improve) and will slow down how you deal with problems because if you suddenly issue a ban now and the user isn't below your magic number, you've just acted against how you said you were going to act. Hello drama! [4/4]
    – Andy
    Aug 31, 2015 at 13:41
  • 1
    @Martijn Lack of traffic isn't really a problem to "solve", it's more of a race against time. Quite honestly, we're getting new users, many of which are asking questions: a good sign. We've placed community ads on sites such as Programmers and Game Dev (Stack Overflow doesn't accept community ads).
    – Zizouz212
    Aug 31, 2015 at 14:49
  • 4
    @Zizouz212 so what is this solution solving?
    – Martijn
    Aug 31, 2015 at 14:54
  • 1
    @Andy Major point: yes, it's more work. It's work that we're willing to do, it's work that we have time to do, it's work we can do well. If it weren't, it wouldn't be happening.
    – ArtOfCode
    Aug 31, 2015 at 16:01
  • 5
    I object to your labeling of reviews as “good” and “bad”. Your classification is purely based on going with the majority. What if the majority gets it wrong? And yes, it happens. Ok, so you'll look out for false positives. But that system isn't very useful since in my experience many bad reviewers will never trigger anything — a common type of bad reviewer systematically picks the majority option to avoid spending the time to think.
    – Gilles 'SO- stop being evil'
    Aug 31, 2015 at 20:20
  • 1
    @Gilles good news, we're not doing it. The system's not being used any more.
    – ArtOfCode
    Aug 31, 2015 at 21:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .