More than 30 caravans moved on to the recreation ground in Moor Road — which lies between Grange Infant School and Grange Junior School and runs alongside Farnborough Town FC — last weekend.
"The sooner they're gone the better. You always worry about your house," said one man who lives nearby in Moor Road.
"Also there's more traffic — they're up and down like yo-yos in their vans running their businesses."
He said the site was popular with travellers, claiming it was targeted at least once a year.
"It's happened so many times before. No doubt it could take a week before they get rid of them," he said.
Mothers who use the play area and whose children go to the schools said young travellers from the campsite had been going into the schools.
A mum, who did not want to be named, said: "One of the boys coming out of school yesterday said children had come into the playground with dogs and were calling them names. It's annoying because you're not supposed to take dogs in there."
Another one said she felt uncomfortable going up there on her own.
"It's not very pleasant wandering up and down there with the kids. It's not so much intimidating but you're conscious of them being there. I wouldn't want to be on my own up there."
She also said she felt uncomfortable being at home by herself at night when her husband works night shifts.
At the time of going to press Helen Lolly of environmental services said Rushmoor Council was doing all it could to deal with the travellers.
"We are taking action to get them moved on as soon as possible. Hopefully we'll get permission from Aldershot magistrates to get them moved on Friday (July 12)."
She said if this was successful the site would be cleared by July 15 at the latest.