Economist Arthur Grimes, a senior fellow at Motu Research and professor at Victoria University of Wellington, has described peple who hold anti-immigration views as "xenophobic."
"Anti-immigration is being xenophobic. Most of the people who are strongly anti-immigration are xenophobic," he told Jack Tame, as part of Q+A's Our Future: Immigration special.
"It certainly doesn't have a negative impact on the labour market and jobs or employment."
However, when questioned whether there were any legitimate economic arguments against immigration, Grimes conceded the one area impacted is house prices.
"If we had a proper decent housing supply through the country, we'd be able to overcome that," he added.
The professor concluded that while our current average intake is doable, surges, not the level of immigration, are what cause issues.
"It's when it surges by an extra 50,000 in a year, that's when housing and infrastructure and things can't keep up."
Demographer Arama Rata argues the immigration system as a whole is racist.
She points to the role of migrant workers on temporary visas in New Zealand.
"These people have no safety net, they can't access any of the resources that we can as citizens. We exploit their labour, then ship them home. So it is racist, it's really unfair," she says.
"If we look at many aspects of our immigration system it's based on nationality, which is the same as having it based on race. So if you were to apply to enter New Zealand through Immingration NZ, on the screen that the workers there see if your risk profile based on what country you are from.
"So this is racial profiling of people based on where they're from and what harm they might do to New Zealand."
Filmmaker and social commentator Oscar Kightley says it shouldn't matter where immigrants come from, but it does.
"I'm sure those people that complain about house prices aren't talking about the Americans that are here buying houses or the people from the UK, its people with 'Asian-sounding names'."
Massey University sociologist Paul Spoonley says New Zealand needs to find a way to address that social anxiety and racism.
He says there needs to be more opportunities for different communities to mix and learn from each other, however he cautions against "cultural literacy" as a deciding factor.
"I would not want cultural literacy to be a criteria for deciding who should come, but we as a community have a responsibility to increase that cultural literacy on arrival."
Spoonley says New Zealanders need to wrestle with the country's racist history.
"Many aren't aware that Chinese migrants couldn't become citizens until 1951. So we've got a long history of racism towards migrants from Asia."