Update in 2016:

Please note this blog is from 2011 during the initial POWER7 days and technology has moved on with faster CPUs and memory in addition there has been software improvements.  I am amazed how many good computer people read this and assume this blog is true to all time!

Virtual Ethernet is faster now but there is also a warning here. Here is an analogy.

• Most vehicles can do 10 MPH (including me on a bike), most cars can do 100 MPH (including my family car) but very few vehicles can do 1000 MPH. That last times ten multiplier is a real massive jump!
• The same issues arise from 100 Mb/s to 1 Gb/s and on to 10 Gb/s = easy, easy and hard.

Multiple Gb/s is possible over virtual Ethernet (especially on POWER8 or later) but you have to work at it. That means network tuning and may be application changes.  Small packets is a killer just due to the number of packets you need and the "transactional costs" of dealing with higher numbers.  You also need much larger buffering at source and target or you run out of room and have to suspend transmission.  There is also problems that are not network related - can you create at the sender the data this fast and process it all at the receiving end.  This is not just a virtual Ethernet issue the same is true for physical networks with hardware assist in adapters and switches.  Finally, take a look at SR-IOV in POWER8 which has hardware but can still allow Live Partition Mobility.

Referring to the below diagram:

1. The network speed between virtual machine A and virtual machine B is purely due to the virtual Ethernet speed - something like 0.5 to 1.5 Gb/s.
2. The network speed between virtual machine A and the VIOS is the virtual Ethernet speed - as above.
3. The network speed between virtual machine A and target outside of the machine via the VIOS SEA is the lower of the virtual Ethernet speed and the physical network speed.
   • If the external network is 1 Gb/s then this it is the limiting factor.
   • If the external network is 10 Gb/s then the virtual Ethernet is the limiting factor i.e. about 1Gb/s.
4. Of course, the VIOS using a 10 Gb/s physical adapter or vNIC can maintain much higher external network speed for ALL the virtual machines as each virtual machine can con-currently be sending and receiving network data at the virtual Ethernet speed.
5. Virtual machine C is not using the virtual Ethernet. If it has a 10 Gb/s adapter or vNIC then is can reach higher speeds to other nodes on the network like for example, 5 to 7 Gb/s. The speed depends on a number of things including
   1. Power CPU cycles available in the virtual machine as running higher speed networks does take considerably more Power CPU cycles for device drivers and interrupts. This is often over looked and is true for virtual or physical networks.
   2. Dealing with the data itself - reading the data from disk or creating it or dealing with incoming data or just saving it to disk.
   3. External network contention - although modern Ethernet switches avoid a lot of packet collisions and resends.
   4. The far end also has to deal with the data - we often hit this limitation when tuning our end of a network without realising it. I often get ask - what is stopping AIX going faster and its not our end that is the issue.
   5. Tuning of the local operating system in flow control, RFA settings, buffering etc. see the "no" commands for lots of options.
   6. Large packet always helps as you then get more bytes per network "transaction" - each transaction costs CPU cycles and interrupts.
6. Converting virtual machines A or B at around 1 Gb/s to a type C virtual machine will get you a jump of 5 to 7 times network speed.
7. The Shared Ethernet Adapter (SEA) on the VIOS does require Power CPU cycles (so don't starve you VIOS of compute time) but does not limit network speeds. The bridging is done at a lower TCPIP stack level for maximum efficiency.