I used to have a real problem with this. I think me becoming a dad, having kids years ago cured me of that. Seriously.
I'm only talking about myself, but I think a lot of the reason I had such a hard time saying no was because at one time I had zero self esteem. I mean, none. So I didn't stick up for myself, and I was so afraid of disappointing people. So much so that I never put my own needs first. Ever. I'm not insinuating anything about your situation...just saying, that's how it was for me. And it's not that I don't care about disappointing friends or family now, oc I do....I think I just know when to stick up for myself and put my needs first when I should.
The last big "no" I said was to my kids last weekend, while we were on vacation at this waterpark. All three of my kids and both of their cousins asked me if they could take the tram, by themselves, a few miles away to the other side. And come back to the cabin when the park closed. There is no fucking way I'd ever say yes to that. They were crazy to have even asked me. No way. Never in a million years.