Is it possible for a man to tell the girl he loved to leave him be, only bcos he wants to work on his marriage?

Sounds like you need to find an attorney and talk about the future. If he made a mistake, and meant that he was sorry, then you still would not be going through this. I think it's time to walk. You have been more than patient, and deserve to be with someone who understands the meaning of trust and loyalty. You should not have to share in a marriage, end of story!