Light of different colours have different speeds in a medium. Therefore, the refractive index of glass (the material of prism) is different for different colours of light and the deviation caused by a prism is different for different colours of light. Violet is deviated the most because in glass, speed of violet is least. Red is deviated the least because in glass, speed of red is most.
Concept: Refraction of Light from a Plane Transparent Surface