Yes it’s a problem. I wish someone had told me that years ago as I was in your position and thought it was none of my business as it was only on the weekend so although I didn’t like the idea of it, it didn’t really affect me. I didn’t ask anyone as I was naive, you’ve made a brave choice to talk about it. Which shows that you have identified it’s a problem early on, I never did. It’s taken years to figure out what’s been going on.
Being realistic you have only been together around a year, I think you know the answer but you want someone else to confirm It. You would be doing the kindest thing for you and him both if you made a stand now, I wish I had before things got so messy, you are lucky that don’t have the worry about sharing a home together. If you made it easier for him now by moving in together when he’s already doing this, what is the incentive to change?
he’s not your child’s father, so if you spent some time apart it only affects the two of you and you could see what is really important for either of you.