The biggest impact will be on those who use Kershaw Grange, at Luddenden Foot, who will have to travel to Hebden Bridge and Halifax instead.
Peter Coles, the council's "older people's champion", said he knew nothing of the changes which are part of a council shake up in day care services. He said the changes were likely to cause upset and inconvenience to many.
"It is sad that there is no public awareness of what's in store, which is where the council often seems to fall short," said local Calderdale Coun Coles "There might still be as many day care places overall but people need these things close to where they live."
The other day-care centre under threat is Savile Close, Halifax, whose users are likely to have to transfer to nearby Higgins Close.
Centres at Hebden Vale, Hebden Bridge, and Lower Edge, Rastrick, will be largely unaffected.
Phil Shire, the council's head of adult services, said talks took place this week with users, their carers and staff to tell them about potential changes.
"The existing number of places for people who are frail or suffering from dementia will be maintained and there is certainly no threat of staff redundancies.
"But there are expected to be savings because we will only be using three buildings instead of five and while some people might have to travel a little further, many others could find they have shorter journeys to make," he said.
The centres are run with the South West Yorkshire mental health trust and Calderdale Primary Care Trust, which provide nurses and occupational therapists.
Mr Shire said the council would be able to run a much more efficient transport service to take people to and from the centres and Kershaw Grange could be used for other things, or sold.
A final recommendation will be made in September and any changes introduced next summer.