A Grade-II listed factory destroyed by fire was an “irreplaceable” piece of history, Victorian Society president Griff Rhys Jones said.
The former Fisons warehouse near Ipswich, which burnt down in a suspected arson attack on Monday, had been on the charity’s endangered list.
The TV star said the building should have been seen as “a magnificent opportunity”, not a “white elephant”.
He added: “We cannot afford to lose our industrial heritage like this.”
The building in Paper Mill Lane, Bramford, dated back to 1858 and until 2003 was used to produce fertiliser by the now-defunct Fisons.
Rhys Jones, who lives in Suffolk, said he had seen pictures of the “sad, sad news”.
“What a pity that more was not done to protect [it]. Far from being a white elephant the Fison building was a magnificent opportunity,” he said.
He said the society had been “right to be worried” when the factory was named in its top 10 endangered buildings list in 2017.
“When the Victorian Society speaks it does know what it is saying,” he added.
The owners of the site, Paper Mill Lane Properties, said they had experienced long-running issues with trespassers on the site.
No-one is thought to have been hurt in the fire, which witnesses compared to a scene from “a disaster film”.
In 2014, a plan to turn the site into a £20m housing and business park was approved by councillors, but work had not begun.
However, that planning permission lapsed on 8 April, a spokesman for Mid Suffolk Council confirmed.