I do feel for you, you’re in an awful situation. This is veering into relationship advice, but you have respected his wishes and boundaries, while he hasn’t done the same for you. Even if I understand where he’s coming from (using at home is almost certainly safer for everyone than him just being… somewhere… out there), you have to decide whether this is the life you want long term.
I’ve had the (relative) luxury of it just being the two of us. It’s been stressful emotionally and financially, of course, but I could hold out because I don’t have a child to worry about.