absolutely there will be a difference on pavement, but thats to be expected. dirt, sand and pavement all hook up very differently.
if i were running mainly pavement, say to a riding spot or something, i might opt for 4 PSI instead of 2 or 3.
i cant think of very many instances though where 6 PSI is needed.
there have been times at hillclimbs that ive actually had to put some air back in the tires because it was so tacky, too much traction was a real possibility.
as you play with tire PSI, gearing and tires more and more you start to gather info about what works and why..