While shopkeepers admit it has been hard to judge the salmonella scare's immediate impact, as sales are typically low throughout the summer, many said they were worried about a backlash by shoppers when the affected products are made available again.
In a week where Cadbury agreed with the Food Standards Agency (FSA) to undertake a comprehensive cleaning of its production lines, company bosses said there was no set timescale for getting the products back in circulation.
An FSA investigation to examine whether other Cadbury products could have been contaminated with salmonella is continuing.
Mike Freke, who owns Triangle Stores in Poole, Dorset, said: "I've read that the investigation is carrying on and if it continues to be in the news then it will hit Cadbury's sales and customers might switch to another brand, or not buy chocolate at all.
"Confectionery makes up about 8% of our sales at this time of year so it is a vital part of our business."
Fellow retailer Kelvin Allen from Londis Late Store, Stoke-on-Trent, added: "I can't say I've noticed a fall in chocolate sales yet but the hot weather has made it hard to be sure as sales always fall at this time of the year.
"When the recalled lines come back on the shelves I think people will be a bit wary about buying them and it's likely to hit sales."