If nothing else, if I don't get into ego-contests or struggles for control with the toxic people, they'll get bored and go away. Right?

Generally, I don't think so. The normal ones will go away. Those you don't care much about. It's the particularly bad ones (psychotic, neurotic, or just plain nuts) that won't and THOSE are the ones you have to worry about.

My 2 cents.