The speed of light in glass is not independent of the colour of light.
The refractive index of a violet component of white light is greater than the refractive index of a red component. Hence, the speed of violet light is less than the speed of red light in glass. Hence, violet light travels slower than red light in a glass prism.
Concept: Refraction and Reflection of Plane Waves Using Huygens Principle - Refraction of a Plane Wave