Merle wasn’t the one who abused daryl, it was his father. Remember back to season 3 when merle tugged on Daryl’s shirt and saw his scars. Merle ran away from their parents and basically left his brother behind.