If you want to use vSGA (Shared graphics) then you should not configure the card for Passthrough.
If you want to use vDGA (Dedicated) then you do not need to load the Nvidia driver on the ESX.
Have a look in this recent whitepaper that explains both:
http://www.vmware.com/files/pdf/techpaper/vmware-horizon-view-graphics-acceleration-deployment.pdf