I'm vegan with an omni husband and vegetarian 2yo son. Our situations sound similar, though I'd describe my husband as more of a flexitarian. He eats vegan when he's out a lot, too, but if he wants a steak he has one.
I hear ya about the inner struggle, especially with my son. Raising him vegetarian seemed like a compromise I could live with then I was pregnant, but any time he has dairy or eggs my heart breaks a little. Right now I just try to come up with alternatives for him, and my husband is agreeable, but I really need to just talk to him about it. If you have luck with this, I'd love to hear! It's so hard.