[j-nsp] iSCSI, fast write, slow read

[Mike Williams]

Hey all,

Firstly, this could very much possibly not be related to any Juniper equipment at all. If so, I apologise in advance.



So.
iSCSI.
4 servers.
Target has a 10Gbps Mellanox Connect-X3 Pro.
Three initiators have 1Gbps I350s.

Writing data, and only writing data, to all 3 initiators runs at ~2.5Gbps.
Exactly what I'd hope would happen.

However, reading data is pegged to ~1Gbps.
In fact, "any" reading of data at all pegs the throughput at ~1Gbps.
100Mbps reading, 900Mbps writing. 500Mbps reading, 500Mbps writing.

Not sure if this attachment will make it through, but attached is a graph illustrating.
Before 11:20 was a bonnie++ run reading and writing at the same time.
After that was ~80 minutes of pure writing.
Finally pure reading.


In the middle is an EX3300.

Hardware inventory:
Item             Version  Part number  Serial number     Description
Chassis                                x      EX3300-48T
Routing Engine 0 REV 14   750-034247   x      EX3300 48-Port
FPC 0            REV 14   750-034247   x      EX3300 48-Port
  CPU                     BUILTIN      BUILTIN           FPC CPU
  PIC 0                   BUILTIN      BUILTIN           48x 10/100/1000 Base-T
  PIC 1          REV 14   750-034247   x      4x GE/XE SFP+
    Xcvr 0       REV 01   740-021308   x           SFP+-10G-SR
    Xcvr 1       REV 01   740-021308   x           SFP+-10G-SR
Power Supply 0                                           PS 100W AC
Fan Tray                                                 Fan Tray

Physical interface: xe-0/1/0
    Laser bias current                        :  7.904 mA
    Laser output power                        :  0.5820 mW / -2.35 dBm
    Module temperature                        :  41 degrees C / 106 degrees F
    Module voltage                            :  3.3420 V
    Receiver signal average optical power     :  0.1781 mW / -7.49 dBm


Juniper branded optics both ends, single-mode fiber connecting them.

It has almost entirely no config at all.
12.3R6.6, and a 9216 MTU configured on all ports as the only config change.


Now, the reason I'm writing here is I suspect the EX.

Reading across a back-to-back 10Gbps connection (same Connect-X3 Pros) goes at the sort of lick you'd expect for a disk subsystem that can sustain >2Gbps of writes.


Am I seeing some sort of buffering issue? When a 10Gbps machine sends traffic to 1Gbps machines.
Or maybe a physical limitation of the 3300?


Closest thing I could find was a discussion about Cisco switches.
https://supportforums.cisco.com/discussion/12483511/performance-issue-10gbs-1gbs-6880-x6800-ia-1521sy0a
I don't see any drops logged, and don't understand Cisco config.


Thanks for any advice!

-- 
Mike Williams