It's possible that it was used in the events that we didn't see, but they never found out that DRJ was behind it. It's also possible that he used it in some events that we didn't see in season 4. There are lots of different ways to explain it. Just go with it.