This is adapted from my talk at Bridge Keepers.

Have you ever gone over the speed limit?

Probably. In an observational study, 7 out of 10 drivers sped in an urban area. At some point, you were likely part of that 7.

Have you heard of how toxic the community for the game League of Legends is? Legendary. (Pun intended.) And you would expect that this is mainly generated by a bunch of bad actors. But it turns out only about 5% of negativity came from trolls – the rest came from normal folks “having a bad day”.

So what is it that causes relatively normal people to behave so badly?

I think there are two major things at play: Normalization and Lack of Punishment.

Normalization

We like to think that we are individuals, unaffected by others. But it’s just not true.

The amazing researcher Dan Ariely (author of one of my favorite books) did a study in which he found that people would change their choice of beer at a brewery based on what the person before them ordered. If that person ordered what they were planning on ordering, they changed their order.

That’s kind of insane. If you want one type of beer, why order a different type? Well, because there’s a norm at play that we should achieve variety to better know what options we have, and our brain automatically kicks in. Even though, as found in the study, people who make these group-motivated choices are generally unhappier with the result.

What I’m saying is: Observing others’ behavior actually changes ours.

Which means if we see someone tweet something this horrible…

…then we think this tweet must be acceptable, because at least it’s a joke! Right? Right??

Types of Norms

There are two types of norms.

 Injunctive norms are basically rules.

 Descriptive norms are the norms we understand from our interactions with others or the actual reality we see in the world around us.

Going back to the speeding example: We all know that speeding is illegal. And we know what the rules are. But the descriptive norm is “eh, it’s ok to speed a little bit”. So we do it without thinking of ourselves as criminals.

The problem with this is that norms are like ivy. Once they’re firmly rooted, they’re very hard to change.

So, how do we create positive norms?

Clear Guidelines

Obvious, but bears repeating. Your community needs to have guidelines, and they should be:

1. Simple. If they’re complex, nobody will bother reading them.

2. In-line. Nobody is going out of their way to find your rules. Put them in-line where possible. Many subreddits do a great job of this:

3. General. If your rules are too specific people won’t bother spending the time understanding what they can and can’t do. (And bad actors may try to find a way to technically obey the rules while causing trouble.)

Consistently Applied Guidelines

Studies show that people are much more likely to obey the rules if they’re consistently applied. This means:

1. Train, train, train your team. Being consistent is hard. You should drill, review past moderations together, do flashcards, whatever you need to do to get consistent across your whole org.

2. Create awareness of implicit bias. None of us wants to think we’re biased, but as shown in the beer experiment above, we often are without even realizing it. And when we are rule enforcers, this can be really problematic. There’s immense evidence that police are biased, and that’s something that has very serious consequences. Are these police racist? No, they’re just operating off biases they may not realize they have. There are plenty of great orgs that help teams work on realizing what implicit biases they have. (A few that have been recommended to me: Paradigm, Project Include, Women’s Leadership Institute.)

Self-Moderation

I think this is one of the most criminally underrated set of tools for driving specific behavior.

1. Display true norm rates. People adjust their habits when they see how others behave (like these college students that drank less once they saw the average number of drinks their peers were drinking).

2. Prime people. Putting people in the right mindset can be incredibly effective. League of Legends found that simply displaying messages before a game (like “Players perform better if you give them constructive feedback after a mistake”) they decreased bad behavior by 11%.

3. Add friction. We are obsessed in the tech world with “frictionless” experiences. But if someone’s action may negative affect dozens, hundreds, millions of people? A little friction can be good. Nextdoor added some additional steps you have to take before posting about suspicious people on their platform, and racial profiling dropped by 75%.

Bad news, though…Guidelines backfire if behavior doesn’t match.

The more litter we see? The more likely we are to litter.

The more we see rule breakers unpunished? The more likely we are to break rules.

Rules are not effective if they’re not enforced.

Especially if there’s a big benefit to breaking the rules. After all, if:

• Attention is the goal

• And negativity generates attention

• And punishment is rare…

…why would you stop? Especially if you can get a book deal from being horrible?

This is the Tragedy of the Commons.

We all know the world would be better if we all obey the rules. But if you personally gain benefit from you personally breaking the rules, you may do it at the expense of others.

Certainty of Punishment

So how do you make these guidelines affective? Create certainty of punishment. If punishment is unlikely, you will continue offending.

The death penalty? Doesn’t look like it actually decreases crimes. Because even though it’s a severe punishment, you’re unlikely to get caught and unlikely to be given this sentence.

Blood alcohol checkpoints? Very effective at decreasing drinking and driving. Because you’re very likely to be caught and punished.

So, how do we create certainty of punishment?

Automation

This is the baseline stuff you should be doing.

1. Blacklist words. Anyone using unacceptable words should automatically be penalized.

2. Spot suspicious behavior. Multiple posts in a short period of time? Similar posts across sub-forums? Shut it down automatically.

Flagging

User flagging is a key tool in the fight against negative behavior. Your users are on the front line and they will always be faster than your team.

1. Make it prevalent. This functionality should be really easy to find, always.

2. Create specific flows. People often struggle with flagging because so many things get flagged that they get overwhelmed. Consider more complex flows depending on what flag was thrown. Marked as annoying? Deprioritize? Marked as racist? Prioritize. Marked as “bugs me”? Implement a clever self-resolution flow like Greater Good and Facebook did.

Reputation Systems

Treat repeat offenders differently. Go pick up Building Web Reputation Systems and figure out what’s best for you.

1. Create visibility thresholds. Have repeat offenders’ posts be less visible to others (which really hurts their desire for attention). Or require people to get special flags to show up to general audiences, like Kinja did.

2. Have reputation affect flag weight. If a repeat offender’s post gets a single flag, weigh that more heavily than a single flag on a good actor’s post.

The punchline: Investing in moderation now saves money later.

I know it’s hard to prioritize spending for moderation, especially when you’re starting out. But “we’ll deal with that issue if it comes up” and “I’m sure people will behave” clearly don’t bear out. And if you wait until you have truly toxic norms and lack of certainty of punishment, it’s going to be way more costly.

Let’s go back to that ivy example. Have you seen what walls look like after you laboriously remove the ivy?

You have to sand those suckers off and repaint.

The equivalent for communities? A whole lot of messaging, a whole lot of banning, and a whole lot of complaints until a new norm is established. Just ask Reddit.

A few notes…

Much of the research here is taking from the hard-to-read but incredibly valuable “Building Successful Online Communities”.

Although I have 10 years of experience in the world of community, I am not an expert at moderation or trust & safety. I’m sure I missed things or mischaracterized things. I would love to hear your insights in the comments!