the closest I think of with regard to map listings is Pipeline, and Peekabooms site, both of which have slowed down a lot (I think peekabooms is dead???).

GIMME MAP DB! =) 

I always thought it was interesting to look at; just to see what was going on for others. 

I honestly don't feel a need for map listings. Most maps will probably get listed and never released, and if you're that keen on keeping people aware of what you're doing then you can update your website on a regular basis or post a message in General Abuse (or both).

Having a database for map downloads would be cool in the short term; in the long term, most of the links will be broken. Had Func_Msgboard been built for it, there could have been a classification system for news items (map releases, tools, reviews, etc) in which case that could have been a nice archive. 

 

post them on your website, or someone can make a MapDB which is always out-of-date and incomplete and hard to maintain. 

I'm trying to archive the contents of Func_ with a web spider and most of it works well. However, it can't seem to parse the view_all_threads and view_all_news pages.

There seem to be some difference between all the other threads and these two. My suspicion is that these seem to be read with the extension .php instead of .php.htm (which might be added by the spider).

Is this intentional or could something be done about it? 

it's not intentional, and i can't think of why that would happen. Also, all pages on func are .php, so the spider might be adding .htm to the ones it successfully downloads. 

most likely true. I've tried three different spiders and all seem to have problems with those pages. In one spider I got an error message that indicated that there was insufficient memory to load one of those pages.

What could it be that requires extra local memory on those pages? I could believe if it was the "General abuse" thread that caused this error but that works fine in all spiders.

I *am* running Win95 but I seriously doubt that there's a problem with virtual memory. In that case it's one of the 64k resource heaps, but I doubt that too, I've got them monitored all the time.

Any ideas? 

Have you tried wget? It's a pretty solid tool. There's a windows port with an acceptable gui for making the bat file. 

tip, I've tried TelePort, BlackWidow and WebReaper so far. I'll take a look at wget. Still, if anyone knows why the others choke on those Func_ pages I'd appreciate a workaround (if there is one). 

those pages probably contain more links than any other page. Maybe your spiders can't store that many links from a single page? 

there are more links than e.g. "General Abuse" which has nearly one link per post (the user link), i.e. several thousands. Just parsing that page takes about 1/2 hour ... 

good point 

another good spider is URLtoys: http://urltoys.gotdoofed.com/ 

tip. Similar to wget it also seems a bit more complex (and probably more powerful) to use than a standard WinGUI variant (of which there seems to be an abundance to select from).

I'd prefer something a bit more simple, I've used TelePort occasionally for a long time and I really like it but for some reason it won't work on those pages.

I'll have to leave it for now, my phone bill's going to skyrocket if I continue these experiments ... 

...worked fine for me (wget -rkp http://celephais.net/board). Ph33r gnu. 

wget another shot. Thanks for the option tip, that seems simple enough.

/me downloads "General Abuse" once more ... 

but it's pretty slow since it appears to get all documents sequentially. Many spiders use multithreading for this. Oh well, you can't have all ... 

I logged into Func this afternoon to see various new posts. The one at the bottom -- and hence the oldest -- was this one:

Painkiller - All 15, New 1

But when I clicked the link, there were indeed two new posts. I loaded the forum index and Painkiller thread less than a minute apart -- at approximately 14:30. The latest post was #16 (yes, 16; and the index still reports only 15 posts in the thread) at 13:55, so I know it wasn't added in the seconds between when I loaded the index and when I loaded the thread; and the Painkiller thread is still listed as fourth from the top.

Wow that was even more unintelligble than I thought. Summary: 1) the thread index didn't report all the new posts, and 2) the thread index displays in incorrect post count for the thread.

I've been suspecting something like this for a few days, but I haven't really had such obvious proof until now. 

I've also noticed for several days that something's wrong with the new post count. 

. 

Scampie's theory: Func_Qmap hates Than. 

this must be related to the database outages we had over the weekend. I think i can repair the data, so i'll try that tonight. 

Is the problem fixed? I still seem to miss some posts. 

No, i didn't get around to it. I'll have time tonight, though.