Unfortunately, I suspect it will get worse before it gets better. Not because it inevitably gets to that state, there are plenty of people who hold down a habit without it taking over everything. But his lack of boundaries around your son is pretty glaring.
Also unfortunately, in my experience he’s right in that getting upset doesn’t help, but there’s precious little that does help. You’ve laid out your concerns, what he does with that is up to him and him alone.