Gems do actually contain trace amounts of metals in them. Some contain iron, which is highly magnetic. Real life peridots are actually magnetic and magnets are actually used to test whether they are fake or real. Also, garnets are actually more magnetic than peridots.