no, it’s no basis for a relationship. Eventually it will lead to him taking it from you without asking.. stealing. If there is no trust there, there is no relationship.
A person with nothing to hide does not get defensive.
Children are loving and will accept most people. Is he providing for your children?
I imagine if he was asked to leave, he would struggle to pay his rent, bills, buy food and support himself. A well paid job and he never has any money. What is he spending it on? You are making it incredibly easy for him to kick back with his habit whilst you support him, feed him and now, financially support him. Darling, you deserve better than this. And the man you fell in love with would not treat you like this. But it sounds like the man you fell in love with is lost to drugs.. same as mine.