More than 20 caravans have been parked on playing fields by the industrial estate in Ivy Road, North Town, since April 6.
As well as causing yet more problems for Rushmoor Council and businesses, it means the only public amenity in North Town is out of bounds to residents.
Rose Alden, who lives in Pegasus Avenue and works in the offices of Perei Group Ltd which overlook the fields, said the travellers were a nuisance for everyone in North Town.
She fumed: "We've complained to the council. They just leave a horrendous sight behind them.
"They're a nuisance in the local shops and pubs in North Lane. They're rude and they always try to bargain with you."
She said she was particularly annoyed because their presence was stopping residents from going about their everyday lives.
"The fields are used by a lot of people. They're used by local football clubs and kids who use them as a play area, and a lot use them for walking their dogs."
Ward councillor Keith Dibble said he had called on Rushmoor to tighten security because it was not good enough for such a vulnerable spot.
"It's tucked away so it doesn't get the same profile as other parks in Aldershot," said Cllr Dibble.
"It's not got the security that other parks like the King George V playing fields have. We want enough money to be spent to make it more secure and less attractive to travellers."
Cllr Dibble claimed officials in the council's parks department had said it was committed to making improvements once the travellers moved on.
John Edwards, Rushmoor's director of environmental services, said the council was doing all it could to get the travellers moved on as quickly as possible.
He said Rushmoor would be applying for an eviction order from magistrates on April 12.
Blackwater Way was also home to a group of travellers this week. Eight caravans were parked at the Geodis site next to Camberley Auto Factors from April 6-8.
The yard will have to be cleared up because the travellers left heaps of rubbish strewn across it. Plastic bags, pots and pans and masses of paper are some of the things left behind.