The E-cam is more than just a "little" bigger. People always seem to think that about the E-cam (partially because that's how Ford Racing advertises), but it just isn't true. Lift is not the only important spec of a cam. Duration also isn't. If I'm not mistaken, the E-cam is a 220/220 duration, which isn't too extreme. However, look at the lobe separation angle: 110 degrees. That's a lot different than the stock 115.5 degrees. A cam with 110 LSA is going to lope, and it's also probably going to cause a little rolling idle.
The roller rockers have absolutely no play in this (assuming a proper installation). The 1.6 ratio is the same as stock, which is good.
I'm also confused by your injector/MAF combo. You said the injectors were upgraded to bigger injectors, yet you are still running a MAF configured for 19# injectors??? If so, that's a big part of your problem right there. You are probably running pig rich at idle, and the ECU is trying to compensate by continuously upping and then once again lowering the idle.
Also, what heads and intake are you running?