Moderating internet forums: What’s smart, not what’s new
August 5, 2003 | Web Designer
25 Comment(s)
I am interested in finding out if it is a lot of work to moderate this public forum. How many people moderate the forum? Who moderates the forum?
Do people have experience moderating web forums? Are there any known procedures for doing so? What is the best way to moderate a forum so that information seems accurate and truthful? At what point does moderating a forum become a misrepresentation of the information sent?
Your feedback is appreciated.
Topics: 3-Star Threads, Science
Moderating a forum is fairly straightforward: knowing what you want, deleting entire threads that aren’t going anywhere, correcting the spelling of the word “it’s,” fixing URLs, deleting individual contributions that fail to advance the thread. It helps to have experience writing and editing (and reading student papers, refereeing journal articles, reviewing manuscripts and grant proposals).
As clearly indicated to potential contributors, we do a lot of deleting–only about half of all submitted contributions survive for more than a month. This doubtless hurts a few feelings but substantially raises the quality of the board. Very few published contributions are edited at all, other than silently to correct spelling, update an URL, or to delete a sour note in an otherwise good answer. Our view is that every contribution to Ask E.T. should advance the analytical quality of the thread. We particularly seek to avoid the chronic internet disease of “All Opinions, All the Time.” The idea is to have an interesting and excellent board on analytical design that serves the content and the readers, not a board logging every attempt at publication. We also are ruthless in deleting contributions with incivilities, rants, taunts, and personal commentary on other contributors.
The forum is named “Ask E.T.,” and my interests are reflected in my contributions, selection of topics, and editorial decisions. Sometimes I don’t answer a question because our contributors have already produced good answers, or I don’t have any idea of the answer or don’t have anything to say just now.
A lot of implicit editing now comes from potential contributors themselves, as they see the style of this board and what good contributions look like. This is a very important point, for as more and more good contributions have accumulated over time, less and less editing has been needed because contributors can tell what it takes to get published here.
We are fortunate in having several star authors, who make smart and interesting contributions. They are occasionally identified as a “Kindly Contributor” to acknowledge publicly my gratitude for their writing.
For some boards, a bozo filter may prove useful by automatically deflecting certain trigger words. My friend Philip Greenspun constructed a filter at photo.net which bounced all those who misspelled the word “aperture,” on the grounds that they did not know much about photography.
It may be a good idea to post some notes about your editorial policies near the place where potential contributors submit their efforts.
Five people are involved in the part-time (very part-time) management of this forum: Dariane Hunt (web designer), Elaine Morse (my design assistant), David Rodriguez (programmer), a very wise anonymous external reviewer, and myself.
So moderating a forum is fairly straightforward and not all that time-consuming–if you have excellent contributors, some editorial skills, a clear idea about what you want, and the ability to make quick final decisions. It takes much more effort to answer questions, start new threads, and add new material to the website.
At this forum, in setting up categories and order of threads within categories, I was concerned with reducing the extreme recency bias of the internet. The idea is that threads have their own history and stand on their own, rather than being embedded in an over-arching and arbitrary time sequence.
Thus it is good to see old threads, when a new response comes in, resurface at the top of the New Responses category. This effect has increased since we stopped accepting any new questions and people have look through the old threads to find a place for their new contribution or question.
Perhaps there are other design methods to reduce the recency bias of internet presentations. The bias is overwhelming. Here is an example: when this board is mentioned at a major site (slashdot, Arts & Letters Daily, Kottke, NYTimes), our logs will show, say, 2000 visits from that site the first day, a 1000 the second, and then, as the mention slips into a screen position reachable only by scrolling, 100 on the third day, and by 5 days, nothing. The quality of the thread didn’t change in 5 days, only its relative position on the referring site.
What then are some ideas on how to present material on at a website in a way that sorts out material by merit and by ideas and not by recency? This includes techniques of forum moderating such as re-opening or re-surfacing older items and bringing them to the top of the relevant page or list.
From Philip Greenspun about forum moderating:
“All user-contributed web content needs pre-moderation”
In the mid-1990s when I started building online communities I didn’t understand why publishers like Amazon pre-moderated all user-contributed content such as comments. The vast majority of users were intelligent and well-meaning and only a small fraction of material had to be deleted. It seemed like it wasn’t worth interrupting the flow of conversation and exchange to ensure that an off-topic posting never saw the light of day. It would be intercepted within a day or so and deleted in any case.
The Manila software that Harvard runs behind these blogs shows the foolishness of my point of view. More than 90 percent of the comments posted to this blog are link spammers trying to increase their Google rank by adding comments to old and forgotten postings. Manila makes it impossible to delete this spams except one by one, each one requiring a several page process of confirmation. In the old ArsDigita Community System we had a “delete all from this user” and “delete all from this IP address” option that made it a lot easier. But in the Age of Spam what we really need is pre-moderation. Maybe there should be an option for a vibrant interactive discussion that content goes live for 24 hours without being approved but otherwise given the small percentage of useful non-spam content it seems that the only answer is that nothing goes public without approval.
Another reason to program in pre-approval only is that eventually the moderators of every online forum find other things to do with their lives. The server doesn’t realize this and soldiers on processing postings. Spammers discover a happy home and the database fills up with crud. Software should be robust to the moderator disappearing and in an Internet that is mostly spam that means approval-required-before-going-live.
Six Apart, which produces the unequaled, open source Movable Type blogging software, publishes the authoritative Guide to Combatting Comment Spam. Alas, like all security (antiterrorism, insurance, police, copyright), there isn’t a one-sentence answer except to say every site must make, and continuously review, its own complex cost-benefit assessment and maintain a layered defense.
In Robert Pirsig’s Zen and the Art of Motorcycle Maintenance he frames his discussion as a focus not on “What’s New” but rather “What’s Best.”
It is subjective and it would require more intervention by the moderator(s) but it would remove the recency bias and SPAM avalanche.
Here is Philip Greenspun’s recent “Comment moderation policy“:
Comments on this Weblog are moderated by Ray Fraser, a kind-hearted volunteer, according to the following policy.
The most valued comments are alternative perspectives. If the posting is about a trip to Kugluktuk, Nunavut, a great comment would start “I took a trip there 10 years ago and my experience was …”.
The least valued comments are reviews of the posting, good or bad. The reader has just read the entire posting. He or she doesn’t need someone else’s opinion that “this was great” or “this was bad.” Reviews make sense in the off-line world where consuming the book or movie happens after reading the review and takes a lot more time and effort. In the online world, the comments are usually read after the item being reviewed has been consumed.
Comments that attack another person’s motivation, intelligence, or character are bad because they degrade the quality of the discussion and discourage thoughtful comments by others. For some reason, human beings often are confident that they can discern the hidden motivation for another person doing or saying something. Trained psychiatrists and psychologists, however, do very poorly at this task, so what hope is there for a lay person?
Cute/clever comments that are off-topic should only be published if they are very cute and clever indeed. Off-topic content breeds more off-topic content.
Attributed/real-name content is preferred to anonymous content.
Due to the fact that 90 percent of the comments posted on the former version of this blog were link spam, all comments are reviewed by the moderator before going live.
See the essay by Alan Jacobs, “Goodbye, blog: The friend of information but the enemy of thought.”
Several, although not all, of the problems mentioned can be eliminated by pre-approving comments before they go live and by regular purging of contributions that turn out not to advance a thread. Thus the focus is always on the quality of the thread for the reader, not on the opportunity for anyone to publish whatever they want. Lost is the spontaneous continuity of back-and-forth conversation; gained is a smarter, coherent, and civil thread over the long run–if there are some good contributors along with a firm editorial board. Thus the model for bloggers is that people are publishing on your board and that therefore you exert editorial control of what is published. This is especially the case on the internet, where there are innumerable opportunities to get anything up on some blog somewhere, or indeed to start one’s own blog.
The director of Princeton University Press, Herb Bailey, once lectured the Editorial Board of the Press that our job was not to be fair but rather to publish good books. Of course a fair process might help to cause good manuscripts to show up to be published, but he had a very strong point, especially since there were many more manuscripts than could ever be published (the acceptance rate of the Press was about 5%-10%).
Compare ET’s kind, even gentle publication policies to this hacker classic by Eric Steven Raymond, How to Ask Questions the Smart Way. Here’s a brief excerpt from the disclaimer:
A rejection slip from a Chinese economics journal supposedly went this way (quoted in the Financial Times): “We have read your manuscript with boundless delight. If we were to publish your paper, it would be impossible for us to publish any work of lower standard. And as it is unthinkable that in the next thousand years we shall see its equal, we are, to our regret, compelled to return your divine composition, and to beg you a thousand times to overlook our short sight and timidity.” From long experience on this board, we have learned never ever to get into discussions with sulky rejected contributors (some become maliciously angry) or even to reply with a bizarrely gracious rejection slip. Our policy is identical to that of nearly all edited forums, beginning with the “letters to the editor” section of The New York Times, Science, Nature, and on down. Publishers have no obligation to publish, acknowledge, or reply to the thousands of unsolicited contributions that they receive. Life’s too short.
Every day we receive partially meritorious contributions that are not published; even some of the contributions of our Kindly Contributors are not published. There are billions of other places on the internet where rejected contributions can be published, unreviewed and unedited. But not here.
Which corporations, government agencies, and other network owners are editing Wikipedia? An excellent New York Times article by Katie Hafner:
The wikiscanner that Niels Olson mentioned in a previous post is an interesting and clever application of happenstance data: who changed the wikipage? I think I have two thoughts on this:
1. People seem either amazed, or entertained, or alarmed that someone with a vested interest in information posted on a public site with open editing privilege would actually change the content. I just checked to make sure that there wasn’t a ‘Rafe Donahue’ entry on wikipedia. There is no entry (yet?) but you can bet that if someone said something about me that was untrue or unbecoming, I would likely be changing that entry as fast as I could (as long as that change was in my best interests; if the lie made me look better, I might be less likely to change it…). Or maybe I wouldn’t care; after all, as my Grandma Emma used to say, “Consider the source.” What would you do?
What am I supposed to conclude if I find that someone has changed information about them? Does this make them evil? I suppose at an absolute minimum I need to find out what was changed. Are corporations and government agencies and other network owners assumed to always be running the spin machine in an unethical manner?
And if I don’t know who changed it, does that make it better? What if the change is done without being about to track the ip address? How can the value or validity of the information (wikinformation?) be dependent upon not who changed it but only upon whether or not we know who changed it?
So, I know that I do not know what to think about all of this.
2. The result of the wikiscanner will not be that people will no longer change their own entries; it will be that they will be smarter about it. In the end all that it will show is that there are still some people who don’t understand that others are watching and that there are other people who have gamed the system.
Should we be amazed or entertained or alarmed?
I actually moderate an internet forum on WatchUSeek, one of the larger watch forums on the Internet (we’ve got almost 15000 registered members with 79000+ threads and close to 475000 posts, with an average hit of some 20000+ hits per day. I moderate the vintage watch forum there. And yes, all we do is discuss watches and other timepieces. Surprisingly, we’re not the biggest: there are other sites that are even busier that us, but we’re nicer. 🙂
The owner of the forum chooses people to be moderators based on maturity, technical knowledge (you’ve never known a geek until you meet a watch geek!) and willingness to be there and answer dumb questions: he vets those folks with existing moderators. Usually people are tapped to be mods when they have been helpful in the forums, have cooled down flame wars and fights by bringing things back on-topic, and generally acting like mature adults rather than loons.
WatchUSeek is fairly stringently moderated: there are no political discussions allowed, nor posting of any photos of weapons or the like. There are clear rules posted and everyone becoming a member agrees to them (at least pro forma). The key is that you have there a team of moderators, working world-wide (literally: we have mods from Australia to Canada and back again, in almost every time zone) who are committed to maintaining two things: high quality of discussion and the ruthless suppression of flame wars, trolling and the like. There are a number of supermods who can zap anyone anywhere; I can do that in my forum, but not in anyone else’s forum.
It works well. Sure, we’ve had our share of problems, but given that you have to be a registered member in order to post anything, you can get rid of any troublemakers by simply banning them (and in the case of static IPs, their IP as well). We deal with spammers the same way: immediate ban on them and their IP. Dynamic IPs won’t be banned, but we’ve hard by and large few problems with folks with dynamic IPs. There’s banning, there’s warnings, there are time-limited bans, etc: we have a number of options for punishment available.
It works because people want to be there and are largely willing to accept the rules to do so.
Clear rules and a clear hierarchy of responsibilities helps. We shut down political discussions ASAP and will delete the threads when doing so; we ruthlessly prune and ban trolls and troublemakers, and the mods all know each other via the forums and respect the decisions that other mods make, but there is also a moderator forum that is open and visible only to the mods for discussions about problem cases and the like. Abusive and threatening members don’t last very long.
Doug Reeves, who runs Van’s Air Force, a big forum about home-constructed airplanes, gives potential contributors an earful about VAF posting policies:
His response to rejected contributors who proclaim themselves victims of censorship is particularly helpful in thinking about forum moderation. Doug’s full statement of forum posting policies is here.
Xeni Jardin (Tech Culture Journalist; Co-editor, Boing Boing; Commentator, NPR; Host, Boing Boing tv) wrote a wonderful essay about forum moderation in her contribution to the Edge World Question Center collection of fascinating answers to the question “What have you changed your mind about?” The complete set of answers may be seen here.
Here is her answer to the question “What have you changed your mind about?”:
For our board, the solution was to eliminate direct posting; every potential contribution, before it goes live, must pass through a non-public approval queue that is reviewed by our editors. IP blocking of spammers, weeds, and trolls has also proved useful.
More on moderating comments at BoingBoing. Teresa Nielsen Hayden, a moderator at BoingBoing posted about BoingBoing’s moderation policy.
Paul Graham’s essay How to Disagree.
Mark Liberman, “A comment about comments”
writes:
Earlier today, someone calling himself (?) Baishui submitted this comment on Victor Mair’s post Burlesque Matinee at the Max Planck Gesellschaft:
“My comments were deleted twice here. Apparently, someone is offended by me saying ‘this incident shows how ignorant the West (and its academics) are of the non-Western world’. What pettiness!”
I deleted this comment, just as I had deleted the same individual’s first attempt, which consisted only of a one-sentence indictment of Western academics, and the second attempt, which added the accusation of censorship.
In an attempt to maintain a reasonable signal-to-noise ratio in the comments section, I’ll continue to be skeptical of comments that lack specific and relevant content. To put this skepticism into context, though, you need to understand something about how comments work in a standard WordPress blog.
Go to link above for much more.
Paul Graham’s Hacker News is a forum for start-up entrepreneurs. If there was ever a forum with a cultural consciousness of their own forum culture, and how to maintain it, Hacker News is it. These are all people trying to build user groups with a sustainable culture. Here are some of pg’s recent thoughts on implementing forum controls:
Rapid banning of trolls seems to be an effective troll guard. It also works well with spam. Trolling and spam are both self-perpetuating problems. Users are ruder on sites where everyone else is rude, and spammers are more likely to submit links to sites they get traffic from. So you can prevent both problems by never letting them get a foothold.
Deletion doesn’t have to be manual, especially in the case of spam. Spammers smart enough to measure the traffic they get from HN quickly give up. And the dumb ones obligingly continue to post from banned accounts and IP addresses. So currently 80-90% of spam is killed by software rather than humans.
Flagging turns out to be a feature that saves a lot of work. So does rate-limiting submissions from newly created accounts (and, obviously, the IP addresses they use).
One general approach I’ve found very useful is not to protect against a certain type of abuse till it arises. Aside from obvious things like not letting people vote more than once, you don’t need much protection when you first launch.
There seems to be an almost bizarre difference between the worst of the net and best: the best forums almost universally have members who use their real names. The worst of course, don’t.
It doesn’t appear to be related to the quality of the carbon-based life form at the keyboard either. Yale Law is abroil with accusations of slander on AutoAdmit.com, a forum for law students (eg, Google for Brittan Heller). Our experience at Tulane Med convinced us, as students, to pull the plug on our own forum, even though it proved to be our best source of information after the storm. People weren’t using their real names.
On our new site, TMedWeb, we have a secure site for reviews of residency programs. Everything is encrypted and password protected (using the university’s directory), but it’s still remarkably sensitive information: these are students who are commenting on the residency programs they apply to, and we enforce real names by publishing their university account aliases (eg, Niels Olson is nolson). The reviewers still don’t shy away from commenting on the quality of programs, but the students provide well composed essays instead of the filth that some of the same people wrote on the old, more public, but pseudo-anonymous forums. The utility of the information is yet to be fully realized. For example, the University of Chicago hospital got some rather negative press recently, due to layoffs, and we had a well written review from a student who had recently spent four weeks there, which provided balance for students considering whether or not to keep Chicago on their Match list for residency.
Ask ET is actually more heavily moderated, but is nevertheless open to the public. It is easily the highest quality public forum I know of.
Paul Graham’s Hacker News allows anonymity but every comment goes live first, and the system relies very explicitly on the community to reward good behavior and punish bad behavior. And, as it happens a lot of people use their real names, even people who might have good reason to not hang their reputation out on a public forum:
pg: Paul Graham, inventor of what the first e-commerce site builder, which he sold for gobs to Yahoo. The ups and downs of Yahoo’s advertising revenue may provide a lot of material for the news, but the Yahoo Store operation continues to provide an underlying cash flow.
paul: Paul Buchheit, inventor of GMail.
dhh: David Heinemeier Hansson, creator of Ruby on Rails
aaronsw: Aaron Swartz. Helped write the RDF specification while in middle school and wrote the RSS specificaton before college.
I think there is a general sense on the Internet that young people in particular favor anonymity and are also more prone to the worst public behavior when they go anonymous. I wonder to what extent this is at least partially a function of self-preservation: the young know they’re young.
It would be interesting to find out how, exactly, young people behave differently on secure forums. Will they use such forums at all? What if we get into a realm of speech-to-text? A 1st grader can read but probably doesn’t type much. We might find out some interesting patterns. High-integrity authentication systems, like LDAP and Kerberos, are becoming more understood and more widely deployed in progressively smaller institutions, so it’s something to keep an eye on. Certainly our recent experience at Tulane Med suggests the upshot can be richly rewarding.
Eliezer Yudkowsky has a good essay encouraging moderators to actively defend their communities: Well-Kept Gardens Die By Pacifism.
A brilliant essay by Chris Clarke:
And the accompanying comments are often delightful.
And, for a thoroughly different, though not inconsistent view, reddit:
Jeff Atwood of the highly popular Stack Exchange (SE) sites has some recent comments on dealing with troublesome accounts:
Our method of dealing with disruptive or destructive community members is simple: their accounts are placed in timed suspension. Initial suspension periods range from 1 to 7 days, and increase exponentially with each subsequent suspension. We prefer the term “timed suspension” to “ban” to emphasize that we do want users to come back to their accounts, if they can learn to refrain from engaging in those disruptive or problematic behaviors.
The above is fine for minor infractions, but for more disruptive individuals it may be necessary to escalate tactics, the first (of three) stratgies mentioned is:
A hellbanned user is invisible to all other users, but crucially, not himself. From their perspective, they are participating normally in the community but nobody ever responds to them. They can no longer disrupt the community because they are effectively a ghost. It’s a clever way of enforcing the “don’t feed the troll” rule in the community.
His post has a few other tidbits:
Suspension, Ban, or Hellban?
The folks at SE have really thought a lot about how to keep their community well run:
Avoid the Streisand Effect – be clear about the reason when suspending an account
They’ve tried to consolidate the rules of everything that’s come before (Usenet, Slashdot, web forums, etc.) into a their software eliminating what doesn’t work and keep what does.
If journals are the original forums, this may be the most significant way to elevate
discourse on the internet: elevate the quality of accessible content.
Further encouragement for authors to at least make their work available online—read the entire piece for further elaboration on each point: