As it sounds like you are aware, Mansfield has a really bad reputation but I'll try and keep this as balanced as possible.
Mansfield is just a typical post heavy Northern industrial town, yes Mansfield is in the east Mids but it has a Northern town feel to it. The town centers pretty dreary with the dominating shopping center looking like a grim brutalist concrete block, the nightlife is fairly dire but about the standard you would expect in a depressed area. To be fair the place has improved from its all time low in the early 90's when there was large amounts of drunken violence and regular day time muggings around the town center by assorted alchy's and junkies - that's largely in the past these days
What you said about the people is the real problem, around the turn of the centuary thousands migrated into the area to work at the pits, when that industry disappeared there was insufficient work for the local population which has led to a massive benefit's dependent culture. Compounding this is the fact anyone with any prospect of a career tends to get out as soon as
As I said the upside to this is cheap house prices, and with good transport links to Nottingham and reasonable links to Sheffield it means you dont really have to have much to do with the town itself.
As an example I bought a 3 bed room semi for just under 60k around 3 years away from the peak of the housing boom. If it means putting up with chavs old and young when I go shopping, for that price, big deal.
Best piece ever written about Mansfield -
Mansfield, in the Heart of Sherwood Forest (1 tree left) is a Busy market town(dodgy cheap clothes for sale). Famous for it's industrial heritage (pollution) mining roots (lot's of men with what looks like eyeliner on but actually coaldust in their eyes), and Robin Hood (who's traditional trade is still carried out with enthusiasm by the local residents) who reputedly roamed the surrounding areas with his merry men (pretending to be disabled people selling household goods for charity) during the middle age's. The town's wealth (new police station and courthouse) was built during the 18th and 19th century on textiles (only women have got jobs), and can be seen today in building's such as the Shoe Co Factory on Sutton Rd (falling down and bankrupt) and the giant Torrey factory in Forest town (Automated and stealing all the local water from it's own boreholes (I can't remember voting for a lower water table and higher water bills). Mansfield has always welcomed people with new religious ideas (Quaker's beaten up in Woodhouse 17th century) Visit the Meeting house (graves underneath Tesco's and Duel carrageway) where religious groups met frequently in times past. During the Jacobite rebellion Bonny Prince Charlie raised his standard in Mansfield and was greeted with the usual Mansfield Welcome (apathy). Bringing us up to date on our short introduction to Mansfield we find the quirky local character in all sections of the community both old, (Turn that Fuckin' music down!) and young (Boys="I'm gonna fuckin' kill you", Girls="take me away from this town and I'll give you and all your mates a blow job, by the way how much money do you earn?".