If Fitz shot down the plane (or caused it to crash), he may not have known at the time that Olivia's mother was on the flight, but he certainly knows now in present day. How could he not? They did a full background check on Olivia and I am sure in the course of her relationship she discussed with him how her mother…

I think, given the reveal in this episode, that what Fitz might be experiencing is guilt over what happened to Olivia's mother (or rather what Olivia *thinks* happened to her mother, I think her mother is still alive and hidden somewhere). He may love her, but what draws him to her, is his attraction and a lifetime…