You walk back to his place together and he invites you in. It is still fairly early, so you accept, but you both agree that it is "just for a little while." You are impressed because his place is really nice - well decorated, but decidedly manly; he has good taste. He kisses you again and you start making out on his couch. You are comfortable with him, and he seems comfortable holding you - you fit well together. Soon his hands move from touching your face to holding your waist and caressing your breasts through your shirt. Before long his hand is down your pants. You are a little drunk so you don't stop him - after all, it feels good, and you are really comfortable with him. But then he starts to unbutton your shirt.
You don't want to have sex with him on the first date, so you know you should stop him; but it is hard to say no - not because you want to have sex with him so badly (though you are actually starting to), but because it seems inappropriate or even unfair to stop him. You don't want to disappoint him and ruin the night, especially when it is just to follow some "rule" your mom taught you. So what do you do? How do you stop him without ruining the end of the date? By the time you've considered what the most tactful way of stopping him would be, he's finished taking off your shirt and bra. Your nipples are in his mouth when you realize that he's actually already unbuttoned your jeans...
I agree that stopping the guy at this point is going to piss him off. There is no question that your suspicion is right in that regard - in fact it will probably piss him off more than you realize. So what should you do now, and when should you have stopped him? There was never a time when it seemed appropriate.
In this situation, you should stop him right away. Better late than never. The annoyance you'll cause him and even the bad feelings you'll stamp on the first date - while important - are less important than not being slutty. True, he will lose some respect for you due to your lack of tact and because you led him on, but he would lose more respect for you if you slept with him on the first date.
The answer to the larger question is a little more complicated. For starters, you shouldn't have had four drinks at the bar (you could have nursed two instead). And then you shouldn't have gone inside with him when you got back to his place. And then you shouldn't have let him put his hand down your pants. And then you shouldn't have let him unbutton your shirt. And then you shouldn't have let him take off your bra. And then you shouldn't... etc. I am not saying you should be a prude, but I am saying that you should be a prude on the first date. With each thing you allow him to do, he feels closer to sex and you cause him more chagrin by then stopping him.
On a first date, I recommend not going in his place. You can have the four drinks at the bar if you can walk home or take public transportation, but otherwise the drinks offer too easy an excuse for needing to go inside to "sober up." If you do go in with him, it isn't the end of the world if you make out. You aren't giving the average American guy mixed signals by making out with him and then saying no to sex. The best stopping point is when he goes to put his hand down your pants or when he goes to take off your clothes - whichever one comes first. All you need to do to stop him is this: without interrupting the kissing, gently grab his hand and guide it away from your pants or shirt, to some other place on your body. Then continue making out as if nothing happened. It is a strong but subtle message. If he tries again, you just repeat the motion. If he tries a third time, you pull back and say "I don't want to do that yet." or something similar. Saying "yet" or "tonight" is key, because it hints that it will happen sometime in the future, and this will entice him (though be careful not to tell him when you'll do it).