Abuse Prevention is Tradecraft (Was: “Como is Infosec” by @doctorow) #ContentModeration Is Not #Infosec

It’s sunday evening; there’s a lot to be said for being a stay-at-home-dad, and honestly I actually like having stuff like meal-prep and laundry to do, after 35+ years of “All Unix Network Security, All The Time” living.

But I don’t like having to write this post. Not least I need to be bringing in the laundry and loading the dishwasher and prepping so my partner can start her 9-to-5 — more accurately online-7:30-to-6 — tomorrow without fuss, whilst I prep breakfast for everyone and begin a day full of Hey Duggee, Bluey, Teletubbies, and other make-cultural-benefit-between-nappy-changes for my glorious 13mo daughter.

But I also don’t want to be writing this because I like Cory, who’s written something extraordinarily ill-advised and misconceived; I owe some small debt to Cory, not only for working vaguely simultaneously for the Open Rights Group together, but also for helping sediment my thinking some years ago that if I ever I had the chance — it seemed unlikely at the time — that I would quit work in order to raise a family, continuing to make impact on the world in time outside of that, rather than as most of my peers had been: wishing that they’d had more time with their kids.

Long story short: Cory wants Facebook tech platform content and community moderation to be more open, to be brought into the daylight, to be where everyone can see how it works, because… well, Cory never exactly gets around to a “x therefore y” reason as far as I can see (?) other than to reasonably point out that having your account shut down without apparent explanation is a pain in the ass, and also to analogise that any amount of secrecy around content moderation is “security through obscurity” — which as any infosec practitioner knows is a “bad thing”.

Except that it’s not — not exactly — a fair comparison; but we’ll come to that.

Cory’s blogpost ends abruptly, as if left hanging by some sudden realisation. I don’t know why or what that is, but leaving the discourse hanging does possibly leave the reader with some incorrect impressions, unless they bother to dig deeper. There’s a couple in particular that I would like to fix, because:

• I used to work at Facebook
• I worked as part of the Site Integrity team
• I designed and wrote software to provide signals for abuse detection
• I worked with and watched the team taking-down a bunch of different kinds of abuse
• I understand how this stuff really works, and…
• …having wrong information flying around in the debate at the moment is really politically dangerous for digital rights.

https://transparency.fb.com/en-gb/policies/community-standards/hate-speech/https://transparency.fb.com/en-gb/policies/community-standards/hate-speech/

Santa Clara Principles?

Cory writes:

That’s why companies like Facebook keep both the rules they apply to community moderation and the tools they use to automate discovery of violations of those rules a secret.

The first half of this is certainly not correct; Facebook at writes at extraordinary length — apparently including historical versions — regarding what activity will result in you being kicked off Facebook. Perhaps he meant to claim that Facebook are lying, instead?

The second half of this bleeds into the subsequent paragraph:

They insist that revealing the policies and enforcement mechanisms will help bad actors who want to harass, defraud or impersonate their users and upload prohibited materials, from disinformation to misinformation, from Child Sex Abuse Material to copyright infringements to terrorist atrocity videos.

Yes, and Facebook’s analysis is correct, and Cory’s critique (“And yet, the same tech giants… routinely use and defend security through obscurity as the only way to keep their content moderation programs on track”) is technically accurate but functionally incorrect, because:

“Information Asymmetry” is not the same as “Security Through Obscurity”

The problem with Security Through Obscurity is (yes) that it’s easily defeated; the usual example is “there’s a spare key for the house, kept under the doormat”, viz: there exists a simple trick which entirely obviates all security mechanisms for the thing being defended.

But Informational Asymmetry (IA) is not the same as STO, and it’s a fundamental of Information Security — or Infosec, since we’re in the land of sexy terminology.

But when you’re in the land of (for instance) anti-abuse, you’re not even in the yes-or-no world of binary truths; instead you’re in the world of aggregate signals:

• IA is knowing that the Russian spammers use a Curl client with a particular TLS fingerprint
• IA is knowing that a troll farm is trying to reduce grammatical language errors by (detectably) posting wholesale chunks of Wikipedia articles as conversational content
• IA is knowing what image-library compression quirks hint that a posting was not sent from an “official” platform client, but instead from some custom bot-farm

So, yes, “revealing the… enforcement mechanisms [absolutely will] help bad actors who want to harass, defraud or impersonate their users and upload prohibited materials, from disinformation to misinformation, from Child Sex Abuse Material to copyright infringements to terrorist atrocity videos”.

And you want to stop that happening?

Cory again:

This is the same failure mode of all security-through-obscurity. Secrecy means that bad guys are privy to defects in systems, while the people who those systems are supposed to defend are in the dark, and can have their defenses weaponized against them.

Okay, maybe I get it; perhaps Cory’s goal is an “accelerationist evolutionary” one, that if we force the big tech platforms to reveal how they are detecting spam then innocent victims whose accounts get closed will obtain some transparency, and the result of all the “bad actors” suddenly getting a free pass towards raising their game / getting better at not being caught, will require Facebook (et al) to once-and-for-all work out a way to “fix” abuse.

Not even regrettably, this is a terrible idea; if anything is going to force Governments around the world to start demanding digital identity cards or some other cure that is worse than the disease, it is this. We should not even go there. We already have (see below) a developing set of principles and an implicit plan to assist people who are victims either of misclassification or of abuse. Being distracted from this towards absolutism, is unwise.

So: Facebook (to name but one) does not keep its rules secret, and it (and the rest of the platform community) is correct and probably wise to be reticent about how it attempts day-to-day to react to the ever changing behavioural “tells” of spam and abuse.

Yes it’s a pain that abusers can infer the (current) rules well enough to know that if they stop just short of referring to a member of an oppressed minority group as [some pre-existing slur] then they can act with some degree of impunity — this appears to be what Cory is presenting as the “key under the doormat” of abuse prevention.

However this is actually an argument for finer-grained, better resourced and (ideally) community-integrated moderation — so the communities themselves can police their own membership — noting in passing that such will of course permit (e.g.) white supremacists to protect themselves from harmful, hurtful ideas such as liberalism, equality and equity.

But the opposite — “perfect” top down control —would be worse.

So: Cory’s blogpost is ill-founded; but perhaps he has a point that platforms at least should be sharing these anti-abuse poker “tells” more transparently amongst themselves and each other, in order to collectively better-prevent abuse?

The sooner we start treating como as infosec, the better. A good first step would be to adopt the Santa Clara Principles, a multistakeholder document that sets out a program for accountable and transparent moderation.

The big platforms already have conferences about sharing the generic — and sometimes the detailed — techniques of abuse-prevention mechanisms with each other; e.g. at the significantly-Facebook-sponsored At Scale conferences with tracks on “Spam Fighting” and “Fighting Abuse” (search for videos, well worth the effort) and of course there is venerable USENIX’s “Enigma” conferences which cover some of the best-of-the-best work in this space.

So when Cory links to, and calls for adoption of, the Santa Clara Principles, I am like… “what?” because:

In 2018, alongside the Content Moderation at Scale conferences

… wait, haven’t we seen that name before somewhere? …

In 2018, alongside the Content Moderation at Scale conferences in the United States, a group of human rights organizations, advocates, and academic experts developed and launched a set of three principles for how best to obtain meaningful transparency and accountability around Internet platforms’ increasingly aggressive moderation of user-generated content.
[…deletia…]

Since 2018, twelve major companies—including Apple, Facebook (Meta), Google, Reddit, Twitter, and Github—have endorsed the Santa Clara Principles and the overall number of companies providing transparency and procedural safeguards has increased, as has the level of transparency and procedural safeguards provided by many of the largest companies.
[…deletia…]

For these reasons, a broad coalition of organizations, advocates and academic experts worked together in 2020 and 2021 to develop this second iteration of the Santa Clara Principles…

Oh, and that explains everything. We’ve gone from a 2018 Version 1 checklist of “what companies need to do in order to be fair to users” to an much-expanded 2021 Version 2 “human rights with design principles” document; and this blogpost is Cory (and perhaps, by extension, the entire EFF) thinking that the best way to get platforms to adopt the new document version is to get people angry about the matter, as-if the document hadn’t come out of the platform community in the first place?

I’m not saying that the new document version is bad — there’s quite a lot in v2 which is a sensible and proportionate evolution of the v1 document, although some new parts are massively onerous and clearly designed to pander to the interests of civil society data scientists who want material with which they can flog the wicked, capitalist platforms in order to justify their salaries:

• The number of times a post was viewed before it was removed. Stakeholders emphasized that the number of views a post received before it was removed, or its virality, is important for understanding the impact a piece of content had before it was removed.
• The timeline for content removal. This includes data on:
  • Time between when a post was published and when it was removed
  • Time before a post that was erroneously removed was reinstated (either as a result of an appeal or as a result of proactive identification of an error by a company)
  • Time between user flagging and response from platform
  • Time for a piece of content to be identified by an automated tool or the company and then removed
  • Time for a piece of content to be flagged to a company and removed by an automated tool.

This is data sought for individual takedowns; there must be millions of these per year, and if nothing else the risks of forensic reidentification of some posts from this data will pose concerns for privacy of the authors; not to mention how to address retrospective actions where a report today causes the takedown of content posted several days, if not weeks previously? With this much data, comes a greater amount of context which will never be available; subsequent punditry and opinion based upon this will be misrepresentative or dubious, at best.

But I digress, because this bit of analysis is immaterial for critique of Cory’s blogpost; instead I’ll just summarise:

1. reticence about how abuse-prevention works is not security through obscurity
2. some reticence about how abuse-prevention operates, helps maintain value of those mechanisms
3. platforms, especially Cory’s bête noire, generally already write at great length about how they judge abusive material…
4. …and, if anything, they risk being criticised for being too verbose about it
5. publishing full details of how platforms detect abusive content will lead to a crisis of abuse occurance, which will further lead to demands for illiberal “crackdowns” and further pursuit of state control of the internet, digital identity cards, etc, to “prevent” the same
6. not to mention: the methods change from week to week, are tweaked from week to week, because people are involved at every level; comparisons with poker are not inaccurate, and yet the v2 document seems not to acknowledge that combatting abuse is a dynamic and diverse pursuit
7. trying to make people angry to get them to beat up the platforms / force the platforms to adopt v2 of something they were involved in writing in the first (and second) place, is not a terribly fair nor charitable tactic

I’d like to see better in the public debate.