Residents living near Belle Vue House in Belle Vue Close, formerly a sheltered housing block, are furious Pavilion Housing Association has not kept them informed about the site's future.
They say the block, which backs on to many homes, is attracting unruly teenagers and it will only be a matter of time before drug addicts and alcoholics move in.
Two residents whose homes back on to the site say there are also rumours the site could be used temporarily to house immigrants or addicts.
"We have been asking for the last three years and then all of a sudden we heard that the last residents moved out last weekend," said one resident, who did not want to be named.
The woman added: "Pavilion says it is going to secure the site but they have only boarded up the ground floor so how long is it going to stay empty? Kids have already broken down the fences.
"People are saying Pavilion could lease the building out to asylum seekers or use it as a resettlement centre for drug and alcoholics."
Residents in Belle Vue Road claim vandals have already got into a back entrance to their gardens from the block.
"Garage windows have been smashed and the fences are continuously being ripped up and broken," another resident said.
A letter was sent to residents last week in response to their demands to find out what was happening.
But residents said the letter did not answer their concerns and they called for a meeting with a Pavilion representative.
Other residents were angry they did not receive the letter.
A neighbour living in North Lane, who did not want to be named, said he would be most affected.
"We've been completely left in the dark," he added.
"You can not tell me they do not know what they are going to do with it. All we want is to know what is happening."
He also said he did not believe the security was tight enough to deter vandals.
"It's a squatters' delight. The electricity is still on and even though the ground floor is boarded up you only need to walk up the stairs and open a window. It is worrying to think that if anyone came into my property I could not defend myself."
North Town Cllr Keith Dibble said it was important for Pavilion Housing Association to regularly consult residents. He said he would be working closely with the association on their behalf.
"We need to know about the short-term management of the site," said Cllr Dibble.
A spokesman for the housing association apologised to those residents who did not get the letter. He said security measures were adequate and they would continue to monitor the site regularly.
"With regard to the future we are still investigating a number of development opportunities.
"We are in discussion with the council and will subsequently submit plans. The likelihood is that it will be used for affordable housing because there is a clear need for this in the area."
He denied rumours that the association planned to lease out the site.