Chateau Saint-Maire is closed for visitors. You can only admire the building from outside. Still worth to stop for a moment and take a look. Castle is located few steps from cathedral So if you are planning to visits you can see both.
We went to this on Rick Steve's walking tour. There was not much to see and construction prevented us from getting a full front picture. The view on the backside of the town was less than other views headed towards the building. Still worth the small walk if you are on that side of old town.