Bankers, who had already been spooked by the Silvergate and Signature fiascos, have started to distance themselves from the potential fallout.