Turns out, the saga may not yet be over. There is a defect in the current version of VirtualBox. I’m using VirtualBox-4.2-4.2.6_82870_fedora18-1.x86_64 right now and there seems to be an issue where IPv6 performance is only ~22Kb/s or so in most of my experiences.
https://www.virtualbox.org/ticket/9380
Hope this helps someone else.
UPDATE:
Disabling GRO seemed to work around the problem for me.
I spent a bunch of time doing “iperf -V -s” between both a VM and a host on the same machine/lan/network interface. The performance one-way would be fast but the other way would be slow with GRO on. Hope this helps you.