Listology Good News and Bad News
Submitted by jim on Wed, 02/08/2006 - 10:00
Tags:
Well gang, the good news is that Listology is going strong. Lots of traffic. The bad news is, well, that Listology is going strong. My ISP informs me it's currently burning through bandwidth at around eight times the rate of other sites on the same plan. With the current AdSense ads (both the old right-side ones and the newly positioned ones) Listology was consistently running a bit shy of break-even, and I personally can't absorb a big hosting fee hike. So I'm going to have to try two things:
- Trim bandwidth: For starters I'm going to try and keep all but Google's search spiders out. I'm also going to make list discussions visible only to logged in users (there will be a handy login link where the discussion used to be). For the really big discussions, I might try an approach where I only load the first few words of each post, and then let folks click on "read more" links for those posts they want to read to keep the initial request size (and hopefully overall size for a given viewer) down. We'll see how that does. I doubt it'll get me the eight-fold decrease I need to stay on the current plan, but you never know how much those spiders eat.
- Increase ad revenue: presupposing failure on thing [1] above, I'm going to have to toy with making the AdSense ads larger and more prominently placed. Many apologies in advance.
We'll see how all this pans out over the next few days. Thanks for your patience and understanding!
Ooooh... I had to login to view discussion. That little change was nicely done. I look forward to hearing what else you've done, and I'm sure you'll announce it.
Thanks! I haven't been able to deploy yet, as my ISP has to get back to me with some baseline data on Listology's load before I deploy the optimizations, but I'll let you know once they're live.
Since August 2004, Listology has become a second home to me (and I'm sure not just to me). So I hope Listology survives. No matter how many google ads occupy the screen, I want to stay in touch with all the wonderful people here around. So that's what I wanted to say...
Would it help anything if I'd put off all the images in my articles, lists, etc.?
Thanks for the thoughts! I'm optimistic I'll be able to keep Listology up, but in the event of a worst cast scenario I have a few ideas for at least letting everyone stay in touch and retrieve their content so it's not lost.
Don't worry about the images. They are hosted whereever you've put them, so don't affect my bandwidth at all.
Maybe you could put up the 'donate' link right below the Other Projects section. It might just help. Or maybe donors get more privilages?
Damn if only it weren't for college fees...I'd gladly donate monthly to Listology. Keep it going strong jim.
I have a few money-making suggestions (perhaps they're not all feasible, but I'm just throwing ideas out right now):
a)Instead of making banner ads bigger and more prominent, have more variety of ads, tailor them to appeal to people who come to listology, and/or (I don't know if this next one is possible) tailor them to appeal to people searching out specific lists.
b)Have a "donations" bar on the side.
c)Offer member service.
d)Start a trading network in which listology takes in a small amount of profit, and people at the site can sell each other movies, music, games, etc.. The lists would increase the profit made this way, as extreme discussion on a certain subject (ie a movie) would increase the number of people wanting to buy, and drive the price up.
e)Allow listologists to become "members", in which you get a large number of new listing options, etc: for a tiny membership fee.
Have you already implemented these changes? I think my lists are loading faster.
All I've done so far is remove the automatic highlighting (both for "what's new" and to highlight search phrases). I can see where that would make a perceptible difference, but hopefully the real change, which I hope to make by Monday, will be even more significant.
In preparation for bigger backend changes (which will hopefully be invisible to you all), I removed auto-highlighting of new items on list pages. When you go to edit your lists, you will see new instructions for applying the highlighting manually, if you are so inclined. I've made it very easy. Auto-highlighting might reappear in the future as a client-side tool (no server load) but that's way low on the priority list. Sorry! I certainly don't like removing features.
why don't you put a donations link?, i think a few people might be able to give something.
Also, try not only ads, but put links to store, fi sombody goes to Amazon or whatever through your add, you get a percentage of the sales.
I've tried donations in the past, but they came nowhere close to what AdSense brings in.
I will look into more affiliate ideas though.
Thanks!
Why not a donations link in addition to the adsense? It doesn't have to be one or the other :)
I would totally donate to help keep Listology going, and I think others would do likewise knowing that you need assistance with the bandwidth.
:-) Thank you, I appreciate the offer(s)! I'll definitely keep it in mind. What I might try is an annual fund drive, if it's necessary.
Thanks again!
Let's assume that I have a single clue (Colonel Mustard.)
If the problem is pulling out comments from a database then I would store every list & comments page as a single text/format file. This way there'd be no assembling of list plus posts every time a list page is viewed.
Lists with new postings/revisions could then be shielded but available in real time by the "click on "read more" links for those [new] posts." New posts could be framed at the bottom to avoid carrying re-heavy re-loads. These active pages would then be "rebuilt nightly" into single files much as the search index is. Or rebuilt every four hours. Or whenever your server gets bored. Old, unviewed, unloved lists could then sit on a shelf in self-contained packages until pulled out and changed.
Or old, unviewed and unloved lists could be culled along with old, unviewed, unloved and/or single-day-pass members. All of that could screw with internal post-targetted links. I'd mourn this loss but would pass on the sackcloth and ashes.
All posts could be shielded/framed at the bottom behind "click on "read more"." Cookies could keep track of recent participants to avoid the load-reload-check again load-check one more time just to make sure load. I would guess that "load[ing] the first few words of each post, and then let[ting] folks click on "read more" links for those posts they want to read " would lead to a heavier demand on your server.
The page would have to reload every time that someone wanted to follow a thread. It would be worse (much worse) for bandwidth and functionality if every "read more" post was shown in isolation. Clicking "read more" and and down a page, back and forth between posts would just lead to a huge amount of churn. I'd rather see "read more" protecting complete threads... perhaps "read more *17 posts in this thread.* Plus a list of posters. Plus character count. Plus warning. Plus gratuity.
I know you say that graphics aren't the principal stumbling block but I know web sites that give a low bandwidth option for seeing more than a preview of a certain article.
If you want to drive ad revenue I would definitely recommend getting the ad links out of the top green space. It's easy to ignore headers and even easier to tune out the small text in them when you're eye is scanning down the page to get to a list. If it were me I'd probably put the ads in the same type and colour as the list itself and place them between the list title and the beginning of the list proper.
This way when someone scans the title the eye (well, "eyes," hopefully) scan past the title and spill over into the ads... not to mention getting a running start in the ads before hitting the list proper. I might also (in addition?) insert ads into the midst of posts once they reach a certain size. If people keep talking about the Dread Speilberg wouldn't you want to be reminded that NetFlix can bring you all of his movies?
I admit it. I often refresh the recent updates page whenever I'm trying to avoid work and keep track of recent exchanges. You could shorten the "all activity on Listology for the past [mmph] hours" to mmph time. I'd be interested in knowing if there was a significant uptick in usage after you enabled infinitely older recent updates... as in "I've been away for two weeks. Let's see what I've missed."
I'm going to give myself plausible deniability for being an idiot by saying that this all was the first thing(s) that popped off the top of my head. Ooh! one last pop. After a suitable period you could ask people as they log in if they want their posts from more than (let's say "two") two weeks ago to remain up. You could even work your way backwards through the threads to trim split ends down to the healthy dark roots of a... *whew!* I'd gladly give up some (all?) of my manic and/or depressive posts. In the Library.
Okay. I'll admit it. This post did not help. With the Lead Pipe.
Good stuff, very much in line with what I was already thinking, which is...
Listology is very read-heavy.
On any given read, all kinds of stuff happens. Header gets drawn, recent updates gets pulled from DB or cache, content is pulled from the DB and lots of formatting is applied (bullets are put on, changes are highlighted, spoilers are spoilerized, search text is highlighted, carraige returns to <br> tags, all of which involves scanning the text multiple times, looking for patterns, and changing the text). The discussion is pulled from the DB, each post is subject to similar format processing as for the list, but it also gets formatted as nested lists for indentation.
On every read of a list.
So I figure, why not do all that on the write instead? Whenever somebody saves a list, or a comment to a list, I'll just call all that rendering code and save the result to the DB. Then a read just involves getting the pre-rendered stuff and displaying as-is.
There are a few complications: like do I save one version with highlighting and another without? What about the "recent updates" box? What about suppressing the discussion display for non-logged in users? The answer to these questions is probably that I leave the Listology "shell" alone, and store three things: [1] the pre-rendered list without "what's new" highlighting, [2] the pre-rendered list with highlighting, and [3] the pre-rendered discussion. Then I just pull some combination of those three things for each read. Highlighting search text will be a casualty.
And I have to hear from my ISP if any of this even matters. Bandwidth is still a concern, even if processing sounds like it might be the bigger issue.
Thanks for the thoughts!
Take that, Professor Plum!
Rereading what I wrote about recent updates I realize that I was unclear. Perhaps I was just confused and that became very clear. What I meant to say was: generate a new recent updates page every mmph hour on the hour, store it (whole and complete) for an hour before generating a new recent updates page.
In the Conservatory... I'd frame out the header/recent updates/site links and keep it static for half an hour. Recent updates might have to move to ten or fifteen items but the borderlands would be set in stone (or 30-minute concrete) and, what is more, wouldn't have to be refreshed as people move about the site.
Once the list(s) are in pre-rendered form is it practicable that you put a third (or fourth? fifth?... header, list, ads, posts) frame at the bottom if there are new posts which are shielded by that "read more" link. This way people could read list and posts (which are at least 24 hours old) and they have the choice if they want to see stuff less than 24 hours old. If there is anything that new, that is.
Graphically this might be difficult to visually integrate new posts at the bottom with the semi-permanent pre-rendered page above. Perhaps this could be fixed by including the previous post in the thread to the new posts in that last frame. That should be fairly easy to match up. It might also mean that only the foamy head of conversation has to take up bandwidth in that last frame. Header, nav bar, list, ads, old posts all get loaded once in their pre-rendered form.
And hey! If you want to bowl a final frame you can open one for writing new posts. This way you can still see the list, all of the posts and the post that you are responding to without any of them having to reload. I'd love this because I'm often trying to integrate stuff from several different posts in my replies.
If I'm interpreting this correctly then you'd like to keep the "what's new" highlighting for lists. This could be achieved by (yet) another "read more" link that pops up the new version of the list down below... with highlights in its hair. Now you can also compare before and after photos without that grey ghost version. Then, when the midnight hour strikes, new stuff gets integrated into the pre-rendered page with highlights in the hair on its head... this page will be revisited in three days to pre-render a non-highlighted version.
Throw in a mandatory log-in to view discussions and you should (ideally... if I have a candlestick) load up and ride:
[1] header/nav bar only once per visitor/visit to the site. This would be pre-rendered every half hour(?)
[2] The list itself which would load only once per visit to that list/page. This would be updated with highlights within every 24 hours of a change and then without highlights after three days... and never pre-rendered again until a change was made.
Shielded from gratuitous loading behind log-ins would be
[3] Posts associated with the list which would load only once upon request by a logged-in member. This members only page would have (if necessary) "read updated list"-[4] and "read new posts"-[5] links.
[4] Updated list with changes highlighted which would be integrated/pre-rendered into [2] every 24 hours. With a final pre-rendering after three days without highlights.
[5] New posts with those cool yellow bullets and the click/edit-able stars which also would be integrated/pre-rendered every 24 hours. With a final pre-rendering after three days.
[6] A reply/respond window which only appears if you'd like to respond to/start a discussion.
This, in my mind's eye, might appear a little balky and problematic to people who are accustomed to bouncing around the site, opening windows willy-nilly. But it might not be. It might encourage people to keep a window open on a list/exchange that is alive for them and stop the cycle of open, close, re-open, lather, rinse, repeat bandwidth suckage. Or you could cookie them to replicate their view upon a return to a list page...
Right in the Billiard Room.
Ah, frames, somehow I missed that originally. My mind balks at them, skips over them, pretends they aren't there. While they can be implemented well, they are often not, and they break the "back" button. Definitely something I'll keep in mind though.
In further talks with my ISP, bandwidth is an issue, but processing time is a much bigger issue. I think (hope!) the caching scheme I'm working on will make a big difference. I hope to roll it out Monday (should be an invisible change to the end user, if all goes well).
That sounds good. And I'd be happy to see search highlighting gone.
Or if you were evil like me, you could write a robot to "click" Listology Adsense ads while constantly changing proxies, once a day.
:-) Google says they watch for stuff like that, and I know they have folks working there smart enough to catch me. Heck, their coffee machine is probably smarter than I am. Stories abound of folks that have gotten kicked off AdSense.
If google's coffee maker is so smart, how come it just accepted a plea agreement of 16 months for tax evasion?
Jim, I'm very unsavvy, computerwise, and so this is probably a fully dopey idea, but, here goes:
How would it be if you made Listology a lists- only site and farmed off the discussion side of things to one of those big free discussion-hosters [smooth terminology bertie] like Msn Groups or Google Groups or that one Yahoo offers. I'm not sure exactly if or how that would work, but....it's completely not useful, isn't it.
Farming stuff out is not a bad idea technically, but I certainly hope it doesn't come to that! Also, I don't think it would be possible to migrate the existing discussions, at least not into the tools you mentioned. I like the brainstorming though.
Thanks for the support folks! I'm still talking with my ISP to pin down exactly what I can do. A few more notes:
Related to lukeprog's observation, I'd actually be better off if Listology were more image-based. All the text processing is very expensive, and is bogging the server down more than the actual deluge of requests, it seems (still trying to pin this down). This may mean I have to trim a feature or two (like highlighting search results in the requested list) and do lots of "pre-rendering" optimizations. I'll find out more soon.
I suspect it's going to be hard to keep out ill-behaving spiders. One approach I'm considering at the moment is to put all of Listology except the home page behind the login screen. I'd want to find a way to keep letting Google through though.
No worries on that "read more on a per post basis" idea. I won't be doing that. I'm going to wait to hear more specifics from my ISP before deciding what to do there.
Anyway, more to follow...
Ah, it's the text processing and not so much the requests, eh? Like, how many pages have to be instantly updated when someone changes a list or makes a post, and scanning the database with searches, etc.? Might it help to add a Google search box to the page in addition to the more refinable searchbox? Perhaps then Google could carry the brunt of searches for those who don't need to narrow their searches by type, genre, etc.
Re: "all of Listology except the home page behind the login screen." Meaning, somebody who finds a list via Google and clicks couldn't actually see the list until they create an account and login? That can't be what you mean...
I'm considering the Google search option (and it's another potential ad revenue stream), but I think that would be a relatively small improvement. Viewing list pages is what constitutes the bulk of Listology activity (which is as it should be :-).
As for the "everything behind the login screen" idea, that is what I was thinking, but I've decided (pretty quickly) that would be a scorched earth strategy.
And since graphics are gracefully sparse, there's little trimming you can do there to save bandwidth. Moreover, you're not hosting big audio or video files that could be moved to a free server like ourmedia.org.
Personally, I'd rather have more ads than have to click "read more" 20 times to follow a conversation. Perhaps you could just load the list initially, then have a single "read more" button to load and display the conversation? Or, the conversation 5000 characters at a time? (Rather than one post at a time)?
Anything to keep registration free. Thanks, Jim.
The fact that you've kept registration free for so long is a tribute to your generosity.
I appreciate anything you can do to keep it that way.
Hope things work out, jim. Your plans sound fine to me.