Upon reflection, Helen Clark remembers Jim as a politician who "stood for a kinder, fairer, better New Zealand" during his time in the political arena.
In a statement released on behalf of his wife, Carole Anderton, she said he passed away peacefully overnight at Cashmere View Hospital in Christchurch.
He was 79.
His death comes soon after he was honoured recently with the insignia for his services to parliament in September.
Mr Anderton held a special relationship with former Prime Minister Helen Clark, becoming her Deputy Prime Minister in 1999 and worked alongside her until 2002.
Speaking to 1 NEWS, the former Labour Leader said she knew Mr Anderton "from the early 1970's" when they campaigned together for the Auckland City Council.
"He was my Deputy Prime Minister for three years, he was a senior administer for nine years of my government and he was a huge contributor to that government," Ms Clark said.
"From the very beginning I saw in Jim Anderton a leader, who would give very bit of energy he had to achieving the goals he wanted to achieve.
"We saw that when he was Labour Party President, we saw it in him as an opposition MP, we saw it in him as Deputy Prime Minister and senior minister.
"I will remember that incredible dedication, hard work and a vision for a better New Zealand.
"Jim stood for a kinder, better, fairer New Zealand. Everything he did was aimed at that. He was a voice for the voiceless, for the marginalised."
Ms Clark said Mr Anderton was one of the few manufacturers who entered parliament, which helped his knowledge of industries throughout New Zealand and understood regions.
"I think his commitment to regional and economic development was very, very significant," she said.
"I think Jim brought a lot to the table over those nine years. Of course with the change in government, something got un-picked but I would hope that this current government would see in his legacy things that could be reactivated, brought on again so that all parts of New Zealand can flourish."
He famously left the party in 1989 over its Rogernomics policy saying "I didn't leave the Labour Party, the Labour Party left me".
He formed the New Labour party and then led the Alliance party which won 10 seats in the 1999 election, which Ms Clark said was a difficult time in their friendship.
"During the time of the fourth Labour government there were many of us who were not happy. I stayed inside and fought the good fight from within, Jim formed his own party but the pleasure for me was when we came back together, prior to the 1999 election and said we will contest this together and we will form a government together when we win. We did and the rest is history. It was a good government."
One memory which Ms Clark holds close is her small family reception that was held in Mr Anderton's home during her wedding to Peter in 1981.
"I would like to express my great sadness at Jim's death and my sincere condolences to Carol and her family and to Jim's children."