it's not intentional, and i can't think of why that would happen. Also, all pages on func are .php, so the spider might be adding .htm to the ones it successfully downloads. 

most likely true. I've tried three different spiders and all seem to have problems with those pages. In one spider I got an error message that indicated that there was insufficient memory to load one of those pages.

What could it be that requires extra local memory on those pages? I could believe if it was the "General abuse" thread that caused this error but that works fine in all spiders.

I *am* running Win95 but I seriously doubt that there's a problem with virtual memory. In that case it's one of the 64k resource heaps, but I doubt that too, I've got them monitored all the time.

Any ideas? 

Have you tried wget? It's a pretty solid tool. There's a windows port with an acceptable gui for making the bat file. 

tip, I've tried TelePort, BlackWidow and WebReaper so far. I'll take a look at wget. Still, if anyone knows why the others choke on those Func_ pages I'd appreciate a workaround (if there is one). 

those pages probably contain more links than any other page. Maybe your spiders can't store that many links from a single page? 

there are more links than e.g. "General Abuse" which has nearly one link per post (the user link), i.e. several thousands. Just parsing that page takes about 1/2 hour ... 

good point 

another good spider is URLtoys: http://urltoys.gotdoofed.com/ 

tip. Similar to wget it also seems a bit more complex (and probably more powerful) to use than a standard WinGUI variant (of which there seems to be an abundance to select from).

I'd prefer something a bit more simple, I've used TelePort occasionally for a long time and I really like it but for some reason it won't work on those pages.

I'll have to leave it for now, my phone bill's going to skyrocket if I continue these experiments ... 

...worked fine for me (wget -rkp http://celephais.net/board). Ph33r gnu. 

wget another shot. Thanks for the option tip, that seems simple enough.

/me downloads "General Abuse" once more ... 

but it's pretty slow since it appears to get all documents sequentially. Many spiders use multithreading for this. Oh well, you can't have all ... 

I logged into Func this afternoon to see various new posts. The one at the bottom -- and hence the oldest -- was this one:

Painkiller - All 15, New 1

But when I clicked the link, there were indeed two new posts. I loaded the forum index and Painkiller thread less than a minute apart -- at approximately 14:30. The latest post was #16 (yes, 16; and the index still reports only 15 posts in the thread) at 13:55, so I know it wasn't added in the seconds between when I loaded the index and when I loaded the thread; and the Painkiller thread is still listed as fourth from the top.

Wow that was even more unintelligble than I thought. Summary: 1) the thread index didn't report all the new posts, and 2) the thread index displays in incorrect post count for the thread.

I've been suspecting something like this for a few days, but I haven't really had such obvious proof until now. 

I've also noticed for several days that something's wrong with the new post count. 

. 

Scampie's theory: Func_Qmap hates Than. 

this must be related to the database outages we had over the weekend. I think i can repair the data, so i'll try that tonight. 

Is the problem fixed? I still seem to miss some posts. 

No, i didn't get around to it. I'll have time tonight, though. 

fixed. 

duder 

Am I just imagining things or is there still an issue with the new posts indicator? I seem to experience both that new posts are sometimes missed and sometimes indicated again although I've already read them. 

I've had the same with post I already had read, some hours later they was indicated as new again. 

is it specific threads, or just random? 

memory, it appears most in the permanent threads. I'm rather sure that I've missed posts in the Mapping Help thread and repetitive new posts in the Screenshots thread.

Sorry to be vague, I wasn't even sure it was actually happening again. I've also checked each time that I've been logged in properly.

A while (some months) ago, I sometimes wasn't logged in automatically and that was probably the cause of not getting new posts indications.