giving up?

How do you know when its time to give up? I have been fighting with this for some time now. I believe that it has to come to point in my relationship with Mike where i need to move on. I do not feel good about it, we have built a family and a home together. We have a picture perfect life but on the inside we are both hurting. I live everyday feeling like mike hates me and i learned yesterday that he feels the same way. He thinks that i hate him. I somewhat think that maybe knowing this information we would be able to accept that and move forward. I just dont see it happening. Mike is a very quiet person, he does not know how to show feelings and basically just doesnt try because he says it feels forced. i am the complete opposite. I wear my feelings on my shoulder and i say everything thing i feel. Strangely enough when we meet we got along great. I thought from day one that he was the one. Shortly after we found out i was pregnant and i decided that i would be the one to show this man love and he would fall in love w me. everything was good, then the girls were born and it just didnt come natural anymore. All the time and love i was putting into this man went to the girls. I loved mike but the girls were more important. nothing but the best was good enough for them. we played the back and forth game for 3 years. i wanted, still want it to work but there is always this doubt, this thinking that all i have put him thru he must hate me. To add to that, he never showed love. Never touched me, never told me and it just came to the point where i feared i was loving someone that did not and would not love me back. I have done this before and was not gonna do it again.
So here i am trying to decide if i should rent an apartment, try to buy a house, where at, what i will need, when will it happen. it sucks knowing that leaving is something i can do but thinking about packing all my things out of my home and taking them to a strange place and try to make it home...it scares me. i had a hard time moving in with mike, harder time moving from the apartment to our house. i cant imagine how hard it will be moving away from all the work i put into this house to make our home. this is trivial but i cant help to think about driving by here in a few years and seeing a new family in my home. Will they take care of my flowers and shrubs that i planted? will they cut down the rose bush i put up in memory of my grandma? Will their kids play in the fenced in yard that my kids played in?
Then i think about how will Christmas feel without Mike. sharing the girls for the holidays...he has no family, will he do anything special for the years, will he get a Christmas tree? who will buy him presents and make him breakfast? it just breaks my heart to think he will spend his holidays alone, sleeping in, eating fast food.
If i only knew what it was that i am doing that is so wrong that makes him despise me? does he despise me or is that something i made up in my head? did he really want to marry me?
God, please guide me heart and life. i know you have a plan for me, the girls and mike. you want all of us to be happy. together or not Lord, i pray that we can get through this with as little pain as possible.