How do you decide when to say something and when to let it go?

If I were in your position, I would ask your father to come to the party separately for his grandson's sake. Ask him politely to stop by just for an hour or so and say hello to his grandson, and have him leave shortly after saying he has to go somewhere. That way, your son will be happy that he came and your mother can stay at the party to give you company.