I haven't given this a whole load of thought, but just wondering how much of a big deal it would be to just use the Q2 .map format but compile it to a Q1 .bsp format. 

I managed to come up with a solution that works...

I've made just the visible faces have the texture that needs to be smoothed. This seems to only work with indoor lighting though. I do hope it can be applied to outdoor lighting too since I don't want to go back to the horrible method of lighting prior to sunlights.

Here's an example that shows the difference between phong shading and normal quake lighting -

https://twitter.com/GavinEdgington/status/675802921334915072 

The pipes have lighting issues if anything but the visible faces have the smooth texture. I made the non-visible connecting faces use a skip texture to avoid these errors. 

Ah, I see what you mean Fifth, sunlight is not phong-shaded. I'll see if I can fix it. 

It's not perfect, but it's a lot better than the standard lighting.

Another bonus point is that now I can wait some more time before developing my own approach. There are lots of things to do� 

Editor support could get messy. It would probably require messing around with FGD files to convert Quake entities to Q2 entities� or convert the Quake FGD to the Q2 specs.

I know about the differences between Q1 and Q2 BSPs, but never thought about looking into the differences between their .map formats. 

is certainly not perfect and I would want to be able to have _smooth work with other brush types like func_group etc. 

@mankrip
afaik .map format doesn't have that many differences, just 3 extra args. brush contents, face flags, and face light level.
tyrlight already parses q2 maps and generates q1 bsps, however it completely discards the extra 3 args (also, no value texture alignment tweaks).
I don't see why you'd need q2 fgd changes at all, you'd still be targetting q1 mods and tools, and editors that support both would likely continue to support both without really caring.
the real issue is 1) editors that do not support quake2. 2) editors that insist on using wal files instead of wads. 3) editors that assume the palette is that of quake2 instead of quake.
for the editors that are mainitained tbh all it requires is for someone to be vocal enough - its amazing how much of a motivator harasment can be.


@ericw
I didn't change anything to do with sunlight that I remember, so yeah, no support there.
sorry about any other issues with it too. I was going to sit on it while someone else found the kinks for me, before making it public.
but, well, open source matra: release early, release often. I'm too lazy for the second part though.
by releasing I hoped to at least save some time for mankrip.
in theory it should be possible to switch the barrycentric coords part to use the entire polygon instead of a triangle (giving a less-triangular/higher-quality apearance), but I just went with what I understand.

@Fifth, there's nothing specific about classnames, it ought to work with any classname (at parse-time, on account of the way it flags texinfos). certainly there's nothing specific to func_detail.
all I can really suggest is to make sure that there's no tiny cracks between faces, because it calculates normals at each vertex and then just interpolates those over the various triangles that form the surface in a similar way to how a gl engine would interpolate normals.
remember that it'll happily smooth around any angle, 20 degrees, 90 degrees, 170... this means you'll likely want to use some other texture for the top/bottom of your mid-pipe cylinders so that they don't cause it to try to curve around those rings. secondly, sunlight isn't supported right now.

if you're looking at the map in fte or dp, make sure you disable any specular first. the texinfos are flagged to say that normals should be interpolated, but engines don't understand that yet, and yeah, I forgot sunlight, and there can still be harsh (often mid-surface) boundaries due to the maths behind the anglescale thing (at one point I was toying with the idea of these surfaces just always using a value of 1). 

I fixed a few bugs that were causing black fringes in my phong test map, and added support for phong+sunlight.

Here is a build: http://quakespasm.ericwa.com/job/tyrutils-ericw-spike/3/artifact/dist/tyrutils-ericw-v0.15.3-7-g05de1ad-win32.zip
and the source code:
https://github.com/ericwa/tyrutils-ericw/commits/smoothnstuff

Hopefully this works on your map, Fifth!

Note, this is based on the previous release of tyrutils-ericw (0.15.3), I'll need to do some merging later on. 

I'd add support for Quake 2 map in Quake games in a jiffy. Would be really easy too. 

That seems to work actually. Very cool. Have you altered the code at all? It seems slightly less smooth than before (but still a lot better than ordinary lighting). 

Seems like whole number integers on vertices or on texture alignment place a huge role in getting the lightmap to look right. Otherwise you get huge black bars or just errors in the lightmap. 

I see what you mean about less smooth / more banding, I messed up something with that last build; trying to fix it.. 

@Fifth here's another build to try, I think it fixes the things I broke in the last one: http://quakespasm.ericwa.com/job/tyrutils-ericw-spike/9/artifact/dist/tyrutils-ericw-v0.15.3-10-g896b821-win32.zip

There is still room for improvement; I still get a few fringe artifacts on some terrain that was rotated 15 degrees, but it's pretty minor and way better than the last build.

@Spike: I switched this build to use SSE; for some reason x87 vs SSE was making a difference in the barycentric coordinates code, with SSE having less artifacts when interpolating points outside the triangles. I will investigate it better at some point :-) 

I will give it a whirl after work. These compilers give me leaks compared to the ones I normally use 

I normally use bjp's compilers. 

Yeah, these are less forgiving than bjp's. Raising the -epsilon parameter to qbsp can help, default is -epsilon 0.0001, try 0.001, 0.01 

seems to crash for me in this build. No idea what's causing it. 

Here's another try...
http://quakespasm.ericwa.com/job/tyrutils-ericw-spike/10/artifact/dist/tyrutils-ericw-v0.15.3-11-g78ca193-win32.zip

I notice extra/extra4 + soft crashes this one. Sorry about all the buggy builds but it'll take a bit to get all of this stuff stabilized and merged. I gotta go map first! 

This one appears to compile but I'm getting some black bars across surfaces even worse than the first build.
It's very likely this is due what I am testing it on (bsp pipe work) and is caused by micro-leaks and things along those lines. I suspect there might not be an easy fix for this other than to reduce detail for smoothed light maps, have smoothed textures use whole number values instead of decimals when doing its calculations? (I'm pretending I know what I'm talking about by this point, from my tests it appears that the closer to a whole number you get the better the smoothing appears to be across a surface). 

In some cases, the position where two vertices of two different brushes meet can become different for either brush due to differences in precision between them during the compiling. This will make the hidden orthogonal face of one of them to pop out, and this will mess up with the smoothing algorithm indeed.

It's one of the extreme cases I had predicted, but it shouldn't happen too often. 

ERROR: Too many vertices ( 65541 > 65535 )

Map also leaks in some mysterious way I haven't deduced yet because there is no .bsp generated due to the error... not sure if they are related...

Map only has 7901 brushes, so it shouldn't be hitting vertex limits I would think?

Any reason why this limit couldn't be raised? 

nm found the leak, and is what causes the vertex overflow... 

Cool - if you need >65k verts, I think you just have to switch to bsp2 with the -bsp2 flag. I will add that to the error message. 

Which is actually a good working philosophy even if you don't want to ship a BSP2. It allows you to work on an unsealed map for much longer.