The enjoyment of a manual isn't about shifting speed, it's about the process of not only moving the shift lever, but coordinating that with your left leg as you work the clutch. It's simply more engaging than twitching your fingers against a steering wheel paddle.
There's a difference between enjoyment driving on the street, and the business of competition driving in professional or even semi-professional racing. Racing is about winning, period, and that's why you see automatic transmissions in F1, because speed and quickness of transitions and changes are everything. When winning can come down to hundredths of a second, shifting speed matters. On the street? It's about the process, the sound, and the engagement with the machine. I don't care about hundredths of a second. Unless you've done it, when you get in the rhythm of it all, it's a lot of fun, and you have no problem concentrating on the drive, because all of it together IS the drive.