The innovative project, which developers Hebden Royd Development LLP say would solve the town's parking crisis by creating 55 new underground public spaces at Garden Street, was overwhelmingly rejected by councillors.
Coun Dave Young said: "This is a wonderful design but it is in the wrong place. It's completely out of character for the town. It's not wanted or needed."
Hebden Royd Town Council will now recommend Calderdale Council, which has final say, refuse planning permission.
Objectors, who have formed the Garden Street Action Group, said the £10 million development, designed by local architects Studio Baad, was not needed because there was no requirement for either more parking or housing.
Anthony Rae, spokesman for the group, said: "The whole rationale for this development is to increase parking but we are confident that an expanded number of spaces isn't necessary.
"Calderdale Council are trying to spook us into solving a problem that isn't there because it is in their financial interest to do so."
He added there were other areas where parking could be expanded, if necessary, in the future.
And the group argued the building, which would include a basement car stacker, would be too overpowering for the town. They said the stacker itself would be too noisy.
But David Fletcher, of Hebden Royd Development LLP, said councillors were betraying the town's radical past by rejecting the plans.
He said: "We need the car parking space, that has been decided.
"This is a radical design but we are simply following in the town's history of radicalism, it would be an exciting addition to Hebden Bridge. To refuse it is simply reactionary."