The cap in the photo is called a kepi. It's French in origin, first appearing as standard issue in the French army in the 1830s. It's not unique to the Civil War. It's been worn by troops and officers in countries as diverse as Japan, Chile, Peru, Denmark, Romania and Greece.