Half the beneficiaries investigated for fraud by the Ministry of Social Development (MSD) have to answer questions about their relationship status.
Being in a marriage-like relationship can mean changes to someone's benefit entitlements - and their payments may end up being reduced.
Figures from the ministry show that in the 2016/17 year, there were close to 6000 fraud investigations completed.
About half of the investigations closed in that same year involved questions about relationship status.
However, of the total number of investigations completed, overpayments were established in just 1800 cases.
And there were only 431 successful prosecutions.
Lawyer Frances Joychild QC said the figures suggested too many beneficiaries were being needlessly scrutinised.
There should be clear grounds before investigators asked people about their personal lives, she said.
"When you are in a vulnerable position of being on a benefit, it is very unpleasant and stressful to have a fraud investigator contact you and inquire about your relationships with other people.
"It shouldn't happen unless there is reason to believe that there is something amiss."
Beneficiary Advocacy and Information Services worker Karen Pattie said the biggest problem was people don't know what being in a marriage-like relationship means.
"Clients, especially single mothers, are too afraid to even date someone or go out for dinner because does this mean a relationship? Does this mean I'll be investigated? If I talk to my case manager, does this mean I have to continue this relationship because I've declared it? Do I ring them and say, 'oh, we're not dating anymore'."
But MSD deputy chief executive of service delivery Viv Rickard said no one's benefit should be affected by going on a couple of dates.
"Can I unequivocally say having a few dates is nothing to do with this organisation and shouldn't impact on anything to do with what people are entitled to."
When allegations were made without supporting evidence, Mr Rickard said it was more than likely no further action would be taken.
However, he said his staff do need to take suggestions of potential fraud seriously.
"Clearly there are a small number of people who will try things on, and the Ministry of Social Development, we administer $24 billion, so I'm not going to open up the gate to the king's gold.
"But we do need to ensure when there are allegations, we consider them appropriately and if there are matters to investigate we will."
The small number of prosecutions the ministry took showed that sort of action was only taken in the most serious cases, Mr Rickard said.
He said the vast majority of Work and Income's 276,000 clients were following the rules and do get in touch when their circumstances change.