I'll be referencing a few games that properly sequence frightening events and, of course, a few games that are not quite so scary. Amnesia: The Dark Descent (review here), Dead Space, FEAR, Silent Hill, and Nightmare House 2 will be on our discussion list below.
Do: Space-Out Enemy Encounters
Humans tend to naturally fear the unknown -- use that to your advantage. The fewer enemies that players, the better. Throughout the entirety of Amnesia there are about ten monsters. Ever. The game attempts to discourage players from even hazarding to look at those monsters by increasing an 'insanity meter;' as the insanity meter rises, the probability of monsters spotting the player rise as well.
Don't: Have a huge amount of enemies come out at one time all the time
Amnesia's evil-goatee'd twin, Dead Space, goes the opposite route: You can rarely go more than ten minutes without a horde of mutant freaks jumping out from every corner. Dead Space doesn't really take the time to strategically build tension, and in a horror game, that's the key. Developers need to aim to build a gamer's paranoia to the point where they think something could be right behind them at any given time, especially high-tension situations (using music and sounds help). Dead Space tries to do this, but ultimately falls short by having constant threats, leading to no tension in moments that require it most. Monsters just don't surprise anyone in Dead Space - it's predictable, and that's bad. Don't get me wrong, it's one of my favorite games of all time - but the problem is that it's just not very scary.
Do: Make the player (nearly) defenseless.
Gamers should dread each encounter to the point that they may not even want to engage -- retreat, hiding, or avoidance should always seem like an option to players, even if one of those actually gets them into more trouble. One of the main criticisms of Silent Hill is the clunky combat system that doesn't allow movement when attacking, but what many critics fail to realize is that - in almost all horror games - you don't play as a trained soldier but as a regular civilian and thus cannot fight as effectively as in other games. A lack of weapons or special abilities will always make players feel like they have to be more tactical, and when we as gamers are forced to sneak around, it's easy to make us jump at a pin-drop.
Don't: Give the player weapons that can clear a room effortlessly.
When F.E.A.R. does do horror, it does it well; however, being a super soldier that can slow down time overrides the fear that the developers worked so hard to create. Being armed to the teeth doesn't help either. When I have a weapon that's bigger than my body, I feel quite safe in my advances - especially if I can slow time when I get the slightest inkling of 'combat music' approaching. FEAR features a weapon called the 'Type-7 Particle Weapon,' a sort of particle-beam-based sniper rifle, which has the following description: "When the beam hits a target, it will strip the target's flesh." Having that kind of weapon, why should the player be scared of anything? Oh - I know! A fleshless monster! They'll never see it coming.
Do: Isolate the player from allies.
Being alone in a harsh environment is one of the best ways to freak people the-hell-out, and Team Silent understands that. Silent Hill's lack of 'friendlies' creates a constant, subconscious thought-process behind each interaction: is this guy going to try to kill me immediately, in five minutes, or at the end of the game? Characters in Silent Hill that aren't insane are very rare, I can only think of six people in the entire series of seven games that don't eventually try to kill the player's character. Or the player. You better watch out.
Don't: Have a squad of soldiers follow the player.
While Nightmare House is "only" a Half-Life mod (granted, our recent overhaul of Skyrim shows how powerful mods can be), it demonstrates very well that having armed escorts following the player-character around takes the terror out of even the best monster design and creepiest atmospheres. Minor spoiler ahead: they are inevitably killed one-by-one, at which point the fear factor returns in full-force, so try to avoid group combat as a whole. It's just not scary if I can rely on my NPC ally to shoot things for me.
GN's Steve and I are working on a couple of articles for the upcoming Friday the 13th (it's only two weeks away!) that will be entirely focused on scary games and their practices, so stay tuned for those! If you have any suggestions or questions - as always - comment below!