It's not just you. I feel the same way about him and his love life too.
Buffy - She was his crush from the beginning and the standard to whom all others are compared to. Willow and Cordelia always knew this and it was always in the back of their heads. At least Wil wouldn't play consolation prize after Buffy rejects Xander in "Prophecy Girl." Cordelia often brought up Buffy as a comparison when questioning how much Xander cared about her.
Cordelia - Definitely a love/hate relationship. It's too bad Xander was still fixated on Buffy to ever give this one a real go because Cordelia was developing real feelings for him. If Xander had felt more that way the relationship could have been something more than just high school romance.
Willow - I doubt Xander will ever feel anything more than a best friend to Willow, even with their flirting during S3. I'm sure he realized he cared about her more than he thought in "Becoming," but I think that kind of love was mistaken for romantic love on his part.
Faith - Although no romantic love was involved, he made the mistake in thinking he and Faith made a connection on an emotional level after their one night stand. That mistake almost cost him his life.
Anya - From the wedding episode, Xander's knowledge of married life from people he knows are generally unhappy. His parents are always arguing with each other, and when he saw that happening in his future with Anya he figured it was best not to start anything before something bad will happen. He had to have these doubts in the back of his head for what's his name's scheme to ruin the wedding to work so effectively. Anya had her own doubts, as revealed in "Once More with Feeling," but unlike Xander she was willing to commit and overcome those doubts. She had more faith in their relationship than he did.