Over the last couple of months, I've taken to riding the Pan without panniers. Particularly on the way to/from work where I can get away with just the top box.
Handling is significantly improved at high speeds with any hint of a wobble above 110 mph disappearing completely. Even with the screen at full height. The downside, I guess, would be that had the lovely chap who drove into me done so when the panniers weren't fitted, the exhaust would have taken the brunt of the damage. Not a nice thought.
I still put the panniers on whenever I go away for the weekend or whiz off to the airport as I can carry what I need for the weekend, a bike cover for leaving the Pan in uncovered car parks and still have room to cargo net a suitcase on the pillion seat if necessary.