Incandescent Outrage, Film at 11

[branchandroot]

You know what infuriates me most about AO3 (today)? The more I read the little things that wranglers anonymous and not are stepping up to tell us, the clearer it becomes that it could work. I don't mean that in a fuzzy procedural way, either, I mean the actual structure of the archive is completely compatible with changing the archive over to canonical, navigable tags and usable fandom hierarchy navigation and not making the wranglers do every damn thing. IT COULD BE DONE RIGHT NOW. Most of the structure is already in place, it's just completely invisible to the users!

It would not take any extra hand-work on the part of the wranglers. It would not require more downtime than any other commit, or break existing functions. The big change would not even be a very difficult bit of code to write! (Terrifying, perhaps, but not difficult.)

How, you ask? Let me tell you, because the top of my goddamn head is about to blow off with the force of my indignation over the pointless ideological stonewalling that's stopping the archive in its tracks!

1) The absolutely necessary first step, on which all else is predicated, is that the public voices of the Archive must speak up to say that Responsible Parties were, after all, mistaken and that the tagging system needs to change. Announce what the changes will be, and a loose timeline of when, so users can take whatever action seems wise to them (eg doing the canonization of their stories themselves so there are no mistakes). Apologize for taking users down a dead-end path for so long and explain the logic behind the upcoming changes (ie readers actually being able to find the authors' stories and wranglers not dying of overwork as the archive grows). Admit that fixing things will inevitably cause some new mistakes, and surface old mistakes. Refrain from any \o/ whatsoever. Do it all again in email.

2) Next step. Make a "request new canonical tag" page. It should be quite simple, as close to one-click as can be. One text-field for each tag type (Fandom, Character, Relationship, Genre/Flavor/Whatever). If Character or Relationship is filled in, a prompt comes up for what Fandom this should be a child of. If a Fandom is filled, a prompt should come up asking for any character/relationships the suggester can think of offhand to populate it with. I'm thinking it should only be visible to logged in users; anything else is spambait. On the wrangler end, this should be equally one-click, as they review requests. Prospective buttons: Approve (immediately creates canonical), Request Review (pops up a flag in whatever task-flow forum exists, asking another wrangler or maybe staff for another opinion), Approve-Needs Forming (creates canonical which the wrangler may alter the phrasing of and optionally pops up a flag asking people to help find synonyms among the non-canonicals to wrangle into the new canonical), Reject (pops up a "reason" form which sends an email to the suggester with the reason filled in; also adds suggestion to blacklist table, with reason; previously blacklisted suggestions do not go through, just pop up a page with the reason for rejection [malformed, malicious, etc.]; the email and this form should both have a link to Support, in case someone wants to argue or get clarification).

3) Next step. Edit the posting form so that Fandom, Character, and Relationship fields can only be populated from the canonicals (possibly re-using the code for selecting a collection name). Fandom(s) remains the only required tag and must be entered first, to create the pool of Character and Relationship options; a warning to this effect should come up if someone clicks into those fields without entering a Fandom. The only other new bit of code required would be a new field for Genre/Flavor/hold a user poll to decide what to call this one. The place where all the No Fandom canonicals will go, at any rate. The Additional tags field can remain as is, in all its freeform glory, perhaps with a note to the effect that Additional tags will not be wrangled, now. Add a prominent link to the "request new canonical" form, and links beside each tag field that will lead to a page of that tag-type for the fandom(s) entered, so people can check what's available instead of having to guess forever. One new note, one new field, four new links, three altered field types. That's it! Test it and send that puppy live. Now all new stories will be in the appropriate format, working off the already existing canonical tag structure and requiring no further wrangling.

4) Adjust import function to look for and set Fandom first, and then try to match any other tags discovered to the fandom's canonicals pool, so as to allow as much of the filling-in process as can be done. Results will probably be about the same as they are now.

5) Write (another) filter sidebar in which only the canonicals show. A fandom index might show Characters, Relationships, and Genre/Flavor canonicals. A character index might show Relationships that character appears in and Genre/Flavor canonicals. A relationship index might also show associated Genre/Flavor canonicals. None of them should show Additional tags at all; instead perhaps there can be a link to the Tags section and a link to the Search form. Revise the in-menu "or" function so that it usefully searches for "character X OR character Y" AND "flavor A OR flavor B" instead of just "anything with X or Y or A or B" which is useless. All of these will be very straightforward mysql queries, instead of monstrous, ass-end-to walker functions. Hold the execution of this until tags are retrofitted.

6) Prep work done! On to the nerve-wracking, if not really difficult, part. Write a query to replace every non-canonical tag id with its canonical version in the story table of the database. Depending on the database structure, this could be as simple as "story.tag_id = tags.canonical where story.tag_id = tags.tag_id and not tags.canonical = ''". Take a good drink to settle your nerves. Clone the story table. Run the query on the clone; this will probably take a few days to get through. Make a news post that The Time Has Come. Take another drink. Shut down the posting form, import any new entries since cloning, swap the names of the clone and the live tables, revive the posting form, now breathe. That should take maybe fifteen minutes, if people prepare beforehand, and that's allowing a margin for a mild case of hysterics or two. If you want to close posting during the replacement process, you won't even have to do the import step. Lo, the archive is running seamlessly on all canonicals! Take another drink.

7) Now it's time for clean-up. Write a query to delete all empty and/or non-canonical tags. Write another to determine all stories with character/relationship meta-tags, the authors' emails, and send them auto-emails informing them that they have meta-tags Y on Story X that should please be re-set to the appropriate canonicals in order for the story to be searchable. Delete these particular meta-tags as they empty. Send general emails to inform all authors that retrofitting is done and they should please check their stories. Warn the wranglers and support people before-hand, because there will invariably be some mistakes showing up and probably some irate users needing to vent. In fact, make a special news post for them to vent in, including any strategies people can think of for easy checking and clean-up, and link to it in the email. Chairs should be on-hand for their volunteers with tea/hard liquor/kleenex/adorable kitten pictures. Schedule this, people.

8) Once the fallout looks to be dealt with, announce completion and success. Now you can \o/. Send the new sidebar live!

9) It's now time to reform the navigation. The Fandoms lists can probably stay as are, at least for now, but every single index page, whether for a fandom, a character, a genre, or whatever else, every index page should show the branch of the hierarchy it is in, as a breadcrumb at the top of the page. For example, selecting "Bishoujo Senshi Sailor Moon" should show, in the breadcrumb "Sailor Moon - All Media Types >> Bishoujo Senshi Sailor Moon". Selecting "Tennou Haruka" from one of those stories should show "Sailor Moon - All Media Types >> Characters >> Tennou Haruka" (or whatever the parent structure is, in that example). Beside the breadcrumb should be a link to the landing page of the fandom. All of these functions and pages exist already, all that is required is to make them visible to the users as well as the volunteers, with a few conditionals to conceal the wrangling links. The public view of the fandom landing page should also have a link to the "request new canonical" form.

10) While you're thinking about it, fix the advanced search, also, so that it has sub-fields for different kinds of tags, and searches for discrete tags as opposed to doing breakage-prone all-field string comparisons.

Congratulations. You now have a working, navigable, professional looking goddamn archive, that can run fast and sleek with as many readers/users as want to come; be proud of yourselves!


Now. That involved only a small amount of new coding, all of it straightforward, and it will fix both server-load and worker-load. The majority of the fix is one query to canonize existing story tags, and a slightly edited form to select new ones, using canonicals and hierarchy that are already established in every case. The rest of it is simply showing users the navigation that's already there. This change-over would not break any existing archive function. It could be nearly seamless. It would even surface things that are currently mis-wrangled but don't readily show it on the front end as the tags stand. And the wranglers would have the far more manageable job of reviewing requests for new canonicals and maybe populating new fandoms instead of trying to make sense of every senseless tag with their hands tied behind their backs. Everything is in place already, to make this work!

It could be done so easily. It could be started right now. WHAT IS STOPPING YOU?

Comments

[kali]

The only real suggestion I have for this is re: relationships. I don't think they should be canonized in the way you suggested(if I'm using the term correctly). I think only the characters should be canonized, and whenever you want to tag with a relationship, you're given the option to select an N, and then input the N characters you have in mind, no matter what canon they're from. After all, 1) what about threesomes and moresomes and 2) there are crossover couples, and I'm sure a million fics spawned by pan-fandom RPs or whatever if nothing else. >.>

[zulu]

This sounds really sensible. The reduction on the wranglers' time especially, and their frustration about not being able to fix obvious typos. Thanks for outlining how it might be done (and including step one, which is the most important).

[skaredykat]

Yes, yes please to the 10-step program! Step 1 is crucial, but seeing that and how it could be done is encouraging. Despite being frustrated that the first sentence of step 1 has been roundly ignored for far too long.

[mitsuhachi]

The thing that kills me, from purely a user standpoint, is that when I post things I WANT to make it easy on the tag wranglers and I CAN'T. Like, it's really a federal fucking secret what the phrasing you want to use for moirallegiance is? Really? So I make my best guess, from batshit-logic that chances between fandoms anyway and based on a structure I'm not allowed to know about. Which then has to have someone come along behind me and fix shit behind the scenes. I do not like enforced helplessness.

*Also--could this be done without step one? Cause that's the only one I really really can't see them doing at all. Like... maybe they could sell it as \o/we're awesome and \o/ \o/ we've figured out how to be more awesome!

[lacey]

I really like the writing you've been doing about Ao3 and the OTW lately. You are the only one I have on my reading page that talks about it with any sort of regularity, and I like that you do so without all the \o/ you get with the party line.

[foxinthestars]

I like it! Sounds like fresh air to me!

Just as one niggle, I would hope that users could go ahead and post a story even if all the canonicals for it aren't in place/approved (having been the first/only person to post in a fandom before), and I do think it would be good to keep waiting/back-and-forth/"retouch it once you hear from us" to a minimum for users where possible, maybe keep the "uncategorized fandoms" area available for stuff that's pending approval...?

But I'm just poking at your thing because it's so very shiny...

BTW, this whole business just called to mind a TED talk I saw some time ago: Sheena Iyengar on the nature of choice, and on Americans' sometimes muddled view that More Choice (or fewer tagging constraints) Is Always Better.

[momijizukamori]

I just want to say I love you in an entirely platonic way for these posts.

[edenfalling]

This sounds brilliant! Therefore, I doubt it will ever happen.

I mean, I try to put canonical tags on my stories whenever possible, but it's hard to stick all the appropriate ones on if you don't know (and can't find out!) what they are. I also use some extremely non-canonical tags sometimes, but that's because I occasionally want to provide information or commentary that doesn't really fit in a summary or author's note -- and/or I would like that information visible in the fandom works page rather than hidden until someone actively clicks on the story link -- and I would be just as happy having that kind of tag roped off as a sort of... hyperlinked additional author's note, maybe? Instead of jumbled in with all the normal searchable tags. Because that kind of tag is not intended to be searchable and it's stupid to treat them as if they are.

[erinptah]

And, hey, I just realized I have more to say.

There are times when I've been really grateful that existing tags weren't just auto-replaced with the canonical. One of my fandoms was really weirdly organized when I got there -- the part that was the biggest pet peeve for me is that two characters with similar names were wrangled together. If all the instances of "Character A" had been auto-converted to "Character A+", there would have been no way to fix it short of going through all the fics and telling all the authors "hey, I see you were writing about Character A, please go back and change all your tags!" As it stands, all I had to do was unhook the existing "Character A" tag and make it canonical in its own right.

That sort of issue is why I'm still in the "give wranglers more latitude" camp, instead of wanting a full revision to a "run most fields on canonicals only" system.

(Although I really appreciate the thought that went into this plan, and the discussion it's generated. It actually gives people something to chew on, in contrast to most of the "our system is just fine as-is because wranglers are awesome, now stop asking for anything different \o/" official ones.)

[sara]

While I'm not quite at the public expression of frustration stage myself, I will admit that I have never understood -- speaking here as someone who's married to a person with a library science degree -- why the belief at the OTW seems to be that some kinds of academic work (e.g. media studies) have great value, but other kinds (e.g. library science, taxonomy) have none, and that there is nothing to be learned from the years and years of work that people have put in to learning to classify shit (particularly written-down shit), we should just reinvent classification from the ground up.

Goodness knows I know a lot of fannish librarians. Were I looking to design a data structure for sorting metadata about a set of objects, I would start by asking them for help.

But yes, I very much get the impression that this is a situation where commitment to an intellectual program is getting in the way of keeping the servers from falling down and going boom. And that you can't design a sensible automated metadata sorting system unless you're prepared to admit that that is the project you are undertaking.

[busaikko]

Interesting reading and hopefully do-able....

This might also make it possible for people to be one-shot tag wranglers: when a new canon arrives, they could sign on to just add in the new canonicals (at least as far as fandom, character names, canon pairings), which would have the advantages of a) being a small but rewarding bit of work (for volunteers with small amounts of time) and b) setting up a framework for when the fic/art starts pouring in.

I really, really want to see the hierarchy and know what the canonical tags are. (For an example, I just wrote a story for a challenge and as there was no pop-up I just guessed as to what the tag should be. So now there's one challenge with at least three different name tags on the first page....)

[sylvaine]

I wish that the people who run the Archive (... NN, right? and how sad is it that I don't even know this?) would just follow this easy step-by-step process. Seriously.

(I presume the Genre/Flavor category would also include things such as kinks and certain fandom tropes (for instance, vampires, werewolves, etc), yes?)

[anatsuno]

I should be running out the door and I’m not - at all - skilled at this on the technical level like you are; I completely agree that the way things work NOW is unsustainable and it needs to change, and I haven’t read the comments (this is my whole disclaimer), but here’s where and why I respectfully disagree with you here - I hope I’ll be articulate enough.

I understand that in other archives since time immemorial the work the archive did, the infrastructure it provided, was to have / relied on having canonical silos. But the AO3 was created precisely with the philosophy that it would support and encourage diversity and not enforce formulations (much) in those so-called canonical things. this was explicitly asked for/debated/decided at the time, I remember it clearly because it was and remains crucial to me.

What I mean is, the fact that I can designate a work of mine as being LotRiPs (fandom) and not Lord of the Ring RPF (fandom) is not unimportant to me - it is one of the basic, fundamental reasons I archive my work at the AO3 when I have never archived it in a fandom-specific or pairing-specific archive before.

What you’re suggesting would totally make AO3 life easier but I can’t agree with your step #3, at all. I wish wrangling would change but I disagree strongly w/ making tags enforceable from the top. Still & forever. The promise of a lack of top-down imposed vocabulary / taxonomy was made, and I want AO3 to keep it.

(aside: that promise is not even implemented enough imo: I am disappointed that while I can - I do - indicate that my fic is a LotRiPS fic on the page itself and in the header, the title of the page in my browser, which also becomes the text of the bookmark if I bookmark the page on Pinboard, say, or the filename if I save the fic, remains the “canonical formulation” - that’s a top-down imposition I resent, not just because I would resent it no matter what, but precisely because it goes against the promise that was made)

The thing is, whether it ruffles coders and librarians’ feathers or not, different ways to format relationship and fandom tags have semantic meaning, sometimes a LOT of it, see many a fandom kerfuffle - so imo users must be able to tag using those formats if we mean to respect them as creators. The whole point of the tag system is to invisibly unify that diversity, not to erase it.

I’m fascinated with sara’s comment about taxonomy having value as work that is looked down upon in the OTW, and I agree that is a problem - work is work, and librarians indeed have worked long and hard on these issues, we might not have to reinvent the wheel. But I think we’re also doing something that has never been done, precisely because we’re not trying to impose (too much of) a top-down decision on semantic differences that are meaningful to a very wide, very varied and very opinionated crowd. As a friend of mine (a coder, too) put it after reading your post: “So they’re proposing to remove the part of the tagging system that’s more sophisticated than any other site in or out of fandom? No.”

What would help more than enforcing CANONICAL TAGS ARE ALL THAT APPEAR is letting people designate synonyms themselves. So that if I tagged my story with a variant tag, it would say “Is this related to X”? Or, “is there a canon tag you want to hook to?” But I could still tag my stories Star Trek: Alternate Original Series instead of Star Trek (2011), damnit. Just let ME wrangle them as I submit my work; allow users to suggest merges like LibraryThing does, etc. But don’t force my hand.

It seems, from what I hear from within, that building infrastructure for interactive support is regularly backburnered - but it could totally address this.

So yeah, there is a lot to discuss, here. Also, I admit, I'm not just loath to see some of the changes you propose, I’m also irritated that the history of the AO3 seems forgotten; that most people seem to think it all was done wrong because omg some people are egotistical/stupid/inefficient and the Org is a slow dim-witted behemoth, and so on and so forth. There are underlying philosophies behind some historical decisions, if not behind all, and there are ethical discussions that were had at the origin of the archive that we all seem to have forgotten entirely, here.

So yeah, there are reasons why I can tag my fic so the header says it’s a LoTRiPS fic, or a Star Trek: Alternate Original Series fic. There are reasons why I need to be able to say “this is a domlijah podfic” and not a “Dominic Monaghan/Elijah Wood” podfic. And these reasons are why I (and a number of others) archive my work at the AO3. They make the computer work difficult, and organizing wrangling the way it was organized has also required + made human work difficult, so we both agree that some things need to change - but that doesn’t mean disappearing diversity is the answer. We can change in other ways.

At least I very much hope so.

[ar]

Oh, God, I want this.

Back when The Playboy Club was on, I thought two of the characters' last names were Beaseley rather than Beasley for some reason. I wrote fic and tagged it as such, and only months later did I realize I'd been reading and spelling the name wrong the entire time. I was mortified, especially as my stories were among the first posted to the Archive for the fandom and thus had probably been seen by pretty much everyone who went looking for fic.

Had your proposed system been in place, I never would have had this problem. I'm a writer, a reader, and a tag wrangler, and these changes would benefit me in all three spheres. Wouldn't it be nice to think they could occur?

[foxinthestars]

A little more curious poking, would you preserve any metatags for (I'm primarily thinking of) characters? I think we'd be better off without the generic names metatags if we could suss them out, but they're also good for groups/teams; for example I wrangle Fushigi Yuugi and have "Seiryuu Shichiseishi" as a meta-tag with all the Seiryuu Seishi who've been tagged for as subtags (oddly enough, "Suzaku Shichiseishi" is the only one of the four that no one's tagged for yet...). Incarnations/aliases of a character might also seem to make sense as subtags to that character. Showing the menu of canonicals in an outline structure using the metatag relationships sounds sparkly, but would it be practical? What do you think?