yes, the fleece runs smaller than the terry and velour usually runs a bit larger than both (in my experience.) getting a full size smaller in the terry might be a bit much so i would just get the same size you wear in fleece. most of my velour pants are M, but i have had both terry and fleece Ms that were too small. i try to get L terry and fleece just to be on the safe side, but i do have several M terry pants that fit fine.
I usually get a medium in all the pants but sometimes the velour is a little big. The fleece is a little bit smaller but stretches out after you wear it for a little. I think you should just get your regular size in terry like jetta said. I dont find where terry stretches out either.