The inverse Faraday effect is a magneto-optical process allowing the magnetization of matter by an optical excitation carrying a non-zero spin of light. This phenomenon was considered until now as symmetric; right or left circular polarizations generate magnetic fields oriented in the direction of light propagation or in the counter-propagating direction. Here, we demonstrate that by manipulating the spin density of light in a plasmonic nanostructure, we generate a chiral inverse Faraday effect, creating a strong magnetic field of 500 mT only for one helicity of the light, the opposite helicity producing this effect only for the mirror structure. This new optical concept opens the way to the generation of magnetic fields with unpolarized light, finding application in the ultrafast manipulation of magnetic domains and processes, such as spin precession, spin currents and waves, magnetic skyrmion or magnetic circular dichroism, with direct applications in data storage and data processing technologies.