I do almost everything organically, i.e. I just build. It's working o.k. so far, but I can understand why someone would want to plan a map's layout. 

my plan is not having a plan! :) 

I have a bad habit of using only my minds eye. The plan or sketch in my head can change a lot whilst I am mapping. But I often think I would do better if I completed a proper planning phase first..... 

very cool concepts zqf.

I usually do not do that; I have an acceptable memory and I use that. I also noticed my layouts tend to be a lot flatter when I use paper sketches.

I realized that it is important to have a central challenge in a level (activate the bridge, activate the slipgate). Opening the exit can also be turned into a challenge.

I design levels around such a central task (or one bigger, one smaller task).

Ideally, the main challenge should be an interactive setpiece, and the player should understand what the task is before being sent off on their button-pressing / key-finding / monster-killing quest.

The finding stuff part can be as long as you like, and you can torture the player along this path in any way you see fit.

There can be any number of sub-tasks (between 1 and 4 would be reasonable, which is why RMQ has 4 standard keys and 4 custom keyitems, for example).

Do a lead in, establish the theme, then show the player the challenge. Typically the way is blocked somehow, and they must remove the obstacle. Then make them go and collect keys or whatever.

It is nice if the map wraps around itself in such a way that you can look into different parts, but not touch until you get there.

Apart from that, it's a repertoire of standard techniques to build stuff that just developed over a couple years. This goes especially for architecture / brush detail; build flat walls first, then break them up... once you get something that looks good, _do_ absolutely shamelessly reuse it everywhere, just vary it a bit. Be cheap and effective. If you built a cool pillar, copy and paste it! It'll become a recognizable feature. Same for door frames, wall supports etc. Try to come up with unique stuff for setpieces two or three times, though.

It is also nice to have parts of the level where you just kill monsters, no switches or keys involved. The map should not ramble on endlessly though, it should be structured. It's nice if there are ambushes or hotspots alternating with less combat.

The level should be largely linear, because players like knowing what to do next, and it makes it easier for the designer to predict the player. However, don't make it utterly linear. Include choices for the player at some points in the map and have little dead ends with goodies in them. Hide goodies everywhere.

A map doesn't need to be huge or extremely nonlinear in order to be fun to play; just reuse some areas or make the player stay in one area longer.

3D combat is good. Flat maps are boring. Slow lifts are bad. Long / frequent cutscenes are bad. Having to wait on a lift or train is bad. Centerprints get annoying quickly. Running around endlessly without completing a challenge is bad.

Quads are fun. Blowing stuff up is fun. Ambushes are fun, if used sparsely.

If you use experimental things that require the player to think, slow down the game etc., consider providing an alternative to them that's also somewhat challenging, but more straightforward.

Platforming can be fun, but failing a jump shouldn't be punished too hard, although it depends on skill level.

Make it easy to get around the map; wind tunnels help to get from A to B quickly.

Returning to an area from a different angle or on a different height is always nice.

It's good to have two or three things in a map that just look interesting, provide an unusual type of challenge etc, just to give it character. Think in terms of hotspots.

Too many switches or keys are bad. Only use them if it makes sense, don't go overboard. Use a unified look for your switches or buttons.

Traps are totally underused. Combine traps (like nail shooters with crushers). Combine traps and platforming (jumping puzzle with crushers). Use monsters as part of traps - a flak ogre can be a great nail shooter replacement.

Ladders are a nice way to traverse the Z axis if you don't have room for a full staircase. Speaking of stairs, reuse them as well. Save yourself the effort of building tons of different stairs. Build one nice set and use it in different configurations.

In general, players will notice some things more than others. You can get away with reusing smaller stuff. It will actually make the level seem more cohesive.

For lighting, I usually stick to one main type of light per area. I like using sky windows. I use a bright sun but a dim sunlight3 with AguirRe's tool. I use minlight during development, but eventually switch it off and rather place more light sources in parts that require them. I stick to a maximum light value of 250 (it gets clamped anyway) and rather use .wait or .delay 5 if I need certain lights to reach further. I only do this on the main lights though, but there are exceptions. Cheat if it looks better.

If something is weak or superfluous, have the courage to cut it. If you can get away with less, consider doing so.

Present simple stuff (door opening etc) as a spectacle. Make them work for it. 

I realized that it is important to have a central challenge in a level

This is the key I believe - an environment is always better if there is a logical theme - Rubicon 2 is a perfect example - it is a giant hydro-electric power station of some sort. This gives a sense of purpose to the world. So to plan the level with what the player is going to do whilst there in mind is even better. 

What Gb says above is hard to default - that's a pretty German-efficient method for building a level, and its hard to find fault with.

I'll play devils advocate and also suggest that building a series of itneresting interconnected areas will get you a decent map, but not the best one possible.

Typically when building a level or sketching it out a hundred ideas will occur of stuff that could be cool or memorable.

The tricky task (and one that seems to bog lots of mappers down) is choosing the strongest few and basing the map around them.

So whilst you can build a functional world, you also need to capture the player's imagination, get them focused and immersed in what you're presenting.

I find this is best done in two ways -

1. Never leave them unsure of what they're doing.
2. Make what they're doing part of the theme, and fun to do.

The first point is pretty straight-forward. It doesn't mean don't allow them to never get lost (although typically the biggest turn off in any map is getting lost) but rather don't allow them to wonder 'now what?' because immediately they're wondering how to finish your level and get it over with.

Better that they wonder 'ok, I killed the dragon and it smashed into the cliff up there, and I need its heart to unlock the gate... how do I get up that cliff?'.

This means you've clearly presented things in an idiot proof manner - players tend to do exactly the opposite of what you'd expect, so you need to plan the flow of the level carefully.

In the dragons heart example this would mean at the very least showing them the dragons heart lock / door before they fight the dragon. For best results they'd see the dragon fly overhead throughout the level a few times, before fighting it.

Which ties into the second point. You want the player to get involved in playing your level and forget that they're hunched over a screen moving a mouse around.

There are many methods for this - storytelling, atmosphere and creating a rich world to explore. All of these (and I'm sure the list is much bigger than those three) boil down to the level design holy grail - immersion.

You want the player so immersed that when they finish, they're bursting for a pee that they've been holding onto for the past twenty minutes.

Once they've gone to the bog they should then be thinking 'I wonder what would have happened if I got all the secrets?' or similar.

As you might have guessed, I always like to have a story with my level design. Typically this story has no direct effect on the level, but it helps me a lot to build it and lets me create areas with purpose - not necessarily 'and this is the castle kitchen' but more 'and here is where the forgetful ogre cook dropped the key in the privy'.

These micro-stories are fun for you to build around and even if the player never catches on, they're still aware (maybe subconsciously) that something is leading them onwards. 

Also, I tend not to sketch levels, which is a bad habit.

This comes from my maps for the past few years either being speedmaps or based off id1 layouts.

My other maps are usually ones that I work on after another designer has got the ball rolling.

One thing I do do, and seems to work very well for me, is keep a pad nearby when I play test my levels, and jot down every little thing that bothers me about it, or every little idea that occurs.

It avoids breaking the flow by alt-tabbing to a text file and means I get closer to whatever vision for the map lies just around the corner. 

The basic idea for the whole map I keep in my head. I've always wanted what I had planned initially in this form and appeared at the end of creation. But it never worked.
Say, the appetite comes with eating. Same for the mapping. Planned to do one thing, but when you start to build, I always get a lot of thought and begin to deviate from the original plan.
On paper, the overall plan will never happen. All in my head. But if I'm away from the computer, I can jot down on paper some individual rooms.
Another feature I never in a hurry. If I need to think about a spot on the map, but I do not know yet how it will look like, then I stop doing. Or make a pause or switch to a different part of the map.
Coherence map can be and is, in architectural terms, but I can not think of a scenario some idea. 

Any creative process is a process - it has no clear end.

Apart from maybe "it's good enough, release it".

It's what I was trying to say before with the ideas jotted down while playtesting. Though my designs tend to fall short of what proper level designers can accomplish. 

in quake mapping i never make layouts on paper. i start from planning the main idea of the map and the general layout in my mind, since i don't think that a complex 3d layout can be made on 2d paper list. but i can do a paper sketching for some certain areas.
but if it goes to mapping for realistic world games (like half-life2 or anything) the paper plan for building interiors is a must for me. 

One thing I do do, and seems to work very well for me, is keep a pad nearby when I play test my levels, and jot down every little thing that bothers me about it, or every little idea that occurs.

Same here, scribbled notes help me keep track of what I need to change.