What should I do if I don't give money to others?

Give if you have money, and ignore him if you have no money. I guess your family still believes in Buddhism and has a good attitude. My family will give him a hard time. He just said a few nice words and gave him money? If he keeps pestering you and hitting you, be tough, because you are a soft persimmon.