tip, I've tried TelePort, BlackWidow and WebReaper so far. I'll take a look at wget. Still, if anyone knows why the others choke on those Func_ pages I'd appreciate a workaround (if there is one). 

those pages probably contain more links than any other page. Maybe your spiders can't store that many links from a single page? 

there are more links than e.g. "General Abuse" which has nearly one link per post (the user link), i.e. several thousands. Just parsing that page takes about 1/2 hour ... 

good point 

another good spider is URLtoys: http://urltoys.gotdoofed.com/ 

tip. Similar to wget it also seems a bit more complex (and probably more powerful) to use than a standard WinGUI variant (of which there seems to be an abundance to select from).

I'd prefer something a bit more simple, I've used TelePort occasionally for a long time and I really like it but for some reason it won't work on those pages.

I'll have to leave it for now, my phone bill's going to skyrocket if I continue these experiments ... 

...worked fine for me (wget -rkp http://celephais.net/board). Ph33r gnu. 

wget another shot. Thanks for the option tip, that seems simple enough.

/me downloads "General Abuse" once more ... 

but it's pretty slow since it appears to get all documents sequentially. Many spiders use multithreading for this. Oh well, you can't have all ... 

I logged into Func this afternoon to see various new posts. The one at the bottom -- and hence the oldest -- was this one:

Painkiller - All 15, New 1

But when I clicked the link, there were indeed two new posts. I loaded the forum index and Painkiller thread less than a minute apart -- at approximately 14:30. The latest post was #16 (yes, 16; and the index still reports only 15 posts in the thread) at 13:55, so I know it wasn't added in the seconds between when I loaded the index and when I loaded the thread; and the Painkiller thread is still listed as fourth from the top.

Wow that was even more unintelligble than I thought. Summary: 1) the thread index didn't report all the new posts, and 2) the thread index displays in incorrect post count for the thread.

I've been suspecting something like this for a few days, but I haven't really had such obvious proof until now. 

I've also noticed for several days that something's wrong with the new post count. 

. 

Scampie's theory: Func_Qmap hates Than. 

this must be related to the database outages we had over the weekend. I think i can repair the data, so i'll try that tonight. 

Is the problem fixed? I still seem to miss some posts. 

No, i didn't get around to it. I'll have time tonight, though. 

fixed. 

duder 

Am I just imagining things or is there still an issue with the new posts indicator? I seem to experience both that new posts are sometimes missed and sometimes indicated again although I've already read them. 

I've had the same with post I already had read, some hours later they was indicated as new again. 

is it specific threads, or just random? 

memory, it appears most in the permanent threads. I'm rather sure that I've missed posts in the Mapping Help thread and repetitive new posts in the Screenshots thread.

Sorry to be vague, I wasn't even sure it was actually happening again. I've also checked each time that I've been logged in properly.

A while (some months) ago, I sometimes wasn't logged in automatically and that was probably the cause of not getting new posts indications. 

it just happened with the mapping help thread. i checked the last 25, since there were no new marked posts there, but madfox's post *is* new. 

as necros said!

About the repeating news I don't remember the specific threads. 

it looked like the postcount field for Mapping Help was at 1899 even though it had 1900 posts. I synced them up, so now it should work. However, i didn't see any other threads with mismatched postcount. So hopefully that was the only one.

I thought this was a one-time thing, but perhaps not. If you see this happen again, please tell me which thread. There may be an actual flaw in code somewhere, or i may need to do additional data cleaning to get everything back to normal.