Log in

No account? Create an account
AO3 Popularity Formula for Fics - alley_skywalker [entries|archive|friends|userinfo]

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

AO3 Popularity Formula for Fics [May. 15th, 2015|05:45 pm]
[Tags|, ]

I'm done with exams (yay!) which means I have ~2 weeks of "vacation" before I have to start work.

There is lots of stuff to do (like...posting May recs...) but I suddenly had this idea of wanting to figure out which of my fics on AO3 are the most popular. I know, that sounds kind of narcissistic, but I'm curious, and I like stats, and doing this on myself is less creepy than compiling information about some random person who doesn't even know I'm doing it, lol (+ you don't get the advantage of someone else's stats page, which is important because "comment threads" vs. just comments and AO3 has trouble with counting when you ask to sort it on a regular filters page, while the stats page tends to be accurate.)

Anyway. my thoughts were such on the matter:
There are ~4 parameters you could use to measure popularity: hits, kudos, comment (threads) and bookmarks. Hits are a largely inflated number, also their value is questionable, because it really only tells you which story seems to be catching the most attention (given, that some of those hits could be coming from recs, of course). Comments are a lot rarer on AO3, especially because there is a quick-and-easy kudos option for people who don't have much to say. So, I'd say comments, like bookmarks, are probably more valuable (than kudos), because they show a ~greater love for a certain story/investment in it. (There are also subscriptions, which I haven't thought about yet.)

So far, I have something like this:
Kudos + (comment threads x 2) + (bookmarks x 2) + (hits/100) = popularity score
Also, I've been rounding the hits number after division to the whole number, always rounding down (maybe I should just round to the greatest whole?).

This seems sensible to me, but IDK. Would it make more sense to not value comments/bookmarks at twice the value of kudos? Should the hit count be left out completely? (I've noticed that leaving in hits CAN make a difference by 1-2 places in a ranking, and that's at the higher end. It gets wankier the lower down the rankings you go.)

[User Picture]From: gracerene
2015-05-16 02:26 am (UTC)
Oo, interesting! I'd definitely say you should include the hits, because I've noticed that some fandoms (like HP) are just less free with their kudos than other fandoms. For example, right now, the top 4 out of 5 of my fics by kudos are all non-HP, but only 1 of my top 5 by hits is non-HP. And I don't think the quality of my writing is that diff in the diff fandoms.
(Reply) (Thread)
[User Picture]From: alley_skywalker
2015-05-16 05:41 am (UTC)
Hmm. I actually have another theory about this. Are the fics that are getting lots of kudos but not a lot of hits (relatively to your HP ones) in small(ish) fandoms? I have this feeling that sometimes people in smaller fandoms (and sometimes rare pairings) will be more likely to leave kudos. Maybe because they are less picky about their reading or more purposeful about encouraging the authors/fics they like because there is so little content in their fandom that they aren't "spoiled" and/or want to make sure that the people who are writing in their fandom continue writing.
(Reply) (Parent) (Thread)
[User Picture]From: gracerene
2015-05-16 06:31 am (UTC)
Actually, I'd say they are pretty decently sized, for example, Stony in MCU & Thilbo in The Hobbit. I definitely wouldn't say they are starved for fic. I think the main reason why they haven't gotten as many hits yet compared to the HP ones, is because they are more recently posted. Though I do think you have a point about the smaller fandoms.

Some people just use the site differently as well. Like for me, if I enjoy a fic enough to read it through to the end, I almost always give kudos. If I really loved, I try to comment & bookmark. If I can't make it through the fic, that's pretty much the only time I don't at least kudo.

Another point about hits, is that I think it goes up if the same person clicks on it multiple times, yes? I know there are several fics that I am rereading right now that I have already bookmarked/commented/kudoed, so even though I reread the fic every few months, hits are really the only thing increasing.
(Reply) (Parent) (Thread)
[User Picture]From: alley_skywalker
2015-05-16 06:58 am (UTC)
New fics are definitely going to be an a disadvantage, at the very least for the first...month maybe? I feel like most fics get the most of their activity in the first month or so of being posted (likely even less for large fandoms) and then the playing field evens out a little more, even though the earlier fics will always have that head start.
(Reply) (Parent) (Thread)
[User Picture]From: ars_belli
2015-05-16 08:47 pm (UTC)
I'd guess that the time-to-activity relation is exponential. Does AO3 have some way of checking this? Short of posting a new fic and recording the stats at given intervals...
(Reply) (Parent) (Thread)
[User Picture]From: alley_skywalker
2015-05-16 10:06 pm (UTC)
No, I don't think they have anything like that.
(Reply) (Parent) (Thread)
[User Picture]From: evelyn_b
2015-05-16 02:30 am (UTC)
Yay for being done with exams! I hope you get to relax.

I turned off my hit counter because it made me too anxious to see people reading my fics and not saying anything. Now all I see are kudos and comments and it's more peaceful.
(Reply) (Thread)
[User Picture]From: alley_skywalker
2015-05-16 05:37 am (UTC)
Yea, I could see that. I've just sort of accepted that lots of people either never get through what they initially start reading or don't really bother to say anything about it.
(Reply) (Parent) (Thread)
[User Picture]From: phresine
2015-05-16 02:12 pm (UTC)
hi we've never actually interacted - I stalked your down LJ after reading your Antonin/Draco fic on AO3, and stayed for your fic recs.

Back in ye olde days when I still lurked the BBC Sherlock anon meme, they used the 1:20 ratio of kudos:hits. One kudos to every twenty hits was the baseline measure of a 'good' fic. I don't know if they still use this system (or if I've even remembered it correctly), but I'm certainly guilty of using it still when deciding to read a fic/judging if my own fic have been successfully executed.

I don't put much stock in comment counts, since filtering fic by most comments always yields lengthy multi-chapters where the comments are mostly, 'so gooood where's the next bit???'
(Reply) (Thread)
[User Picture]From: alley_skywalker
2015-05-16 07:45 pm (UTC)
Hmm that's interesting. And also a pretty high bar, it seems like. But I've certainly hear similar sentiments when it comes to judging fic quality at any rate.

You're right that with WIPs it get a little iffy on comments. It does show that people keep reading, so I think it's valid as a quality measure, but maybe not popularity as it is likely the same people coming back/commenting over and over again.
(Reply) (Parent) (Thread)
[User Picture]From: ars_belli
2015-05-16 04:07 pm (UTC)
What about using hits as your baselinecalibration parameter? So for each story, find kudos/hits, bookmarks/hits and comment threads/hits. I'd be very interested to see whether there were trends between the ratios. If you just have an arbitrary weighting, your popularity score is very sensitive to the weight coefficients, as you've found out. With the ratios, you're able to measure different expressions of popularity for the same story in a way that lets you compare to the rest of your stories.

(Sorry, the maths major in me is getting very excited. If only I had enough fics to make my stats meaningful!)

If you've written enough fics, it would make a lot of sense to do this by fandom... Heck, I'm sure there's a PhD's worth of interesting data on statistical trends in fandom!
(Reply) (Thread)
[User Picture]From: alley_skywalker
2015-05-16 08:08 pm (UTC)
This is interesting! I mean, the coefficients aren't completely arbitrary, but I am making some assumptions here, of course, e.g. the value of comments vs. kudos.

What you are suggesting would work well if we're looking at quality, it seems to me. As in, how many people chose to/felt compelled to leave positive feedback on a fic. But it takes out the variable of how much traffic each story gets. So what you're suggesting -- (kudos/hits) + (comment threads/hits) + (bookmarks/hits) = score -- would be a good way to measure quality or "critical response" but as far as popularity, I'm not sure. I mean, it depends on how you define popularity, but I'm going off the assumption that the amount of traffic/exposure a fic gets is still at least somewhat important. In the case of what you're suggesting, a lot of traffic/exposure may actually depress the fic's score (unless all of those hits are converting into kudos at a high rate).

Are you aware of fandomstats? They've got a lot of conversations about fandom and statistics by people who know a lot better what they're doing than I do, lol.
(Reply) (Parent) (Thread)
[User Picture]From: ars_belli
2015-05-16 08:43 pm (UTC)
In an ideal world, popularity and quality would be the same thing, right? I don't quite see the distinction (here), because you're still measuring the fandom's opinion of your work. What I'm trying to get rid of is the people who view your fic once, but don't return to it again. How many times have you clicked on a link to a promising fic and thought it was mediocre?

Measuring traffic alone is definitely still a useful measure. But that involves a whole new level of complexity! For small fandoms, you still have to compare to every fic in that fandom (and only in that fandom). In big fandoms, if you have a fic of a certain pairing, you'd have to compare that fic's hits to every other fic with the same pairing, in order to give it a popularity ranking. I assume genfic opens up an even bigger can of worms because people are soooo liberal with character tags.

I suppose what my waffle is trying to say is that measuring popularity always implies a relative measure, so you have to pick your comparisons carefully to end up with a meaningful statistic.

I had no idea about fandomstats, thanks!

Edited at 2015-05-16 08:43 pm (UTC)
(Reply) (Parent) (Thread)
[User Picture]From: alley_skywalker
2015-05-16 10:32 pm (UTC)
I understand what you're saying. But I'm measuring popularity among my fics, not my fics vs. fics in the fandom. So, for me the fact that a fic in a more popular pairing or in a more popular fandom will get more traffic/exposure is something I expect and am trying to factor IN rather than out. If I was measuring my Harry Potter fics against the popularity of other Harry Potter fics, it would make (more) sense to use quality determining factors (like hits:kudos ratios) only.

Here is my issue. Imagine we have large Fandom A and small Fandom B with a fic in each.
Fic in fandom A has 20 kudos. Fic in fandom B has only 5 kudos. Looking at this, you would say that fic in fandom A is more popular (as compare to fic in fandom B).
BUT imagine now that fic in fandom A has 100 hits and fic in fandom B has 15 hits. The ratio of kudos:hits for the fic in fandom A is 1:4. Ratio for fic in fandom B is 1:3. So, by this measure, fic in Fandom B is more popular. But that doesn't jive, does it? How can it be more popular if only 15 (vs. 100) people have read (tried to read, anyway) it and only 5 (vs. 20) liked it enough to leave a kudos?

ETA: So, the question, I guess, is why not use just kudos or kudos/comments/bookmarks without the hits entirely? It would still favor large fandoms i the same way. Which, now that I'm thinking about it, maybe is a way to go after all. I'm just somehow attached to the idea of factoring in hits anyway. I feel like there HAS to be some intrinsic value there too...

(sorry, I know I've kind of used you as a sounding board here <_<)

Edited at 2015-05-16 10:38 pm (UTC)
(Reply) (Parent) (Thread)
[User Picture]From: ars_belli
2015-05-31 12:14 pm (UTC)
Imagine if you only wrote Death Eater fic. Then it would make sense to compare your hit counts, kudos etc. directly between your fics, right? But you don't, so to make a meaningful comparison, somehow you have to "cancel out" the effects of the different fandoms.

The ratio of kudos:hits for the fic in fandom A is 1:4. Ratio for fic in fandom B is 1:3. So, by this measure, fic in Fandom B is more popular.
Sort of? Fandom B fic is more efficient at gaining popularity. For all the people who've read it, 33% enjoyed your fic, compared to 25% in Fnadom A. (But in both cases, kudos-leavers probably account for some non-negligible portion of the hit count because they are repeat readers.)

How can it be more popular if only 15 (vs. 100) people have read (tried to read, anyway) it and only 5 (vs. 20) liked it enough to leave a kudos?
We know nothing about the make-up of the hit counts in both cases. So in some sense, hits are a poor cross-fandom measurement, because we can't assume that the enthusiasm for giving kudos/comments is the same across fandoms. What if you measured your hit count as a percentage of the entire fandom's? In some sense that would measure how popular your fic is relative to that one fandom's activity, which you can then use to compare between your own fics.

The questions that we have to answer are things like:
* Is a fic popular if everyone reads it once and only once?
* If not, then does a unique-reader count have any meaning on its own?
* If we can make hits meaningful, how many repeat hit-counts do we assume from kudos-givers? comment-leavers? bookmarkers?
etc etc etc.
This is a really annoying Bayesian problem, because the priors are all so nebulous!
(Reply) (Parent) (Thread)