Should I break it off with my boyfriend who I just met in college about a month ago?

I met this guy in March and he told me he had a son and I accepted that and we started a relationship real fast. Now, it's summer and I told him I may not come back next semester but I really know that I am. Anyway, we get the talking and I ask him how does he feel about me and he says he likes me but he doesn't think he could ever love not just me but any other female because of his past. He has dated some crazy girls and while he was growing up he was taught that no one comes before his baby mama and once you get married there's no marrying someone else and having a stepmom or stepdad. But, he knows that Im starting to get feelings for him and he's telling me that he doesn't feel the same way. Now, should I just let this go or just work with it because it is still early and things could change later on. I know what I want I think. LOL, I want someone who wants to be with me and be happy. Is that too much to ask for? Am I overreacting?