I found out that was the issue with my Cuz as well. We had a lot of other things going on at the time, and I always assumed those were the issues. The fact that we are family was a major factor for her, but I never knew it. The difference was that I was disappointed, and we tip toed around it, and it was awkward for many years, but I was NEVER mean to her, nor did I rub it in her face with a GF or wife. We always look each other in the eyes, and I would never walk out of a room and leave her alone. She has a BF that EVERYBODY EXCEPT ME has told her she can do better. They have been together for a LONG time, that is HER decision to figure out if she can do better. The guy does have a tendency to rub people the wrong way, but he has never been that way to me, and even if he did, it is still her life and her choice.
With just the information you have given here, I'm going to call "sour grapes" on his part. This, to me, indicates that GF or not, he is not mature enough for you to consider picking up where you left off yet. Should he not be in a relationship at some point, you would want to take this into consideration before you get any more emotional investment in this situation.