You mention you have two children? I think you need to shift your focus and consider the impact and potential dangers to your children. If he owes alot of money (dealers) they wouldn’t think twice about coming to your house regardless if kids are there or not.
Lavish lifestlye? Did you know he was a dealer while going on your holidays and living this new life? If you did I would urge you to go and get some support yourself and discuss this with your family and friends to gain some perspective here.
He in no way sounds stable enough to be around children end of story.