The supermarket originally applied to build the store on the Unthank Road in September 2004. Council planning officers had recommended approval for the latest application, but planning committee members went against the recommendations and rejected the plans.
In a heated council chamber, councillors expressed concern over a number of aspects, including an increase in traffic, too few parking spaces and poor access for delivery lorries.
Norwich independent retailer Nigel Dowdney told C-Store: "This is an excellent result. It should be a real boost to other campaigns and shows that councils are finally starting to stand up against the power of the supermarkets."
It is not yet known whether Tesco will appeal the decision. A spokesman said: "We will await the planning report and consider our options."
In Slough, Tesco is being forced to sell a site it acquired when it bought a former Co-op store. The Competition Commission has concluded that the acquisition of the Uxbridge Road store, which took place in 2003, "reduced competition and choice in the market for grocery retailing" in the town.