Here's some suggestions...
I am not sure what you mean by leaving the "limbs some room to breathe" ? All bows operate most efficiently (and generally more quietly) at the max setting, which means with the limb bolts turned all the way down tight (don't get out a torque wrench, off course, but just hand tighten all the way.) If you are shooting with the bolts backed out more than one turn, you could be at 65# or lower, which would affect the arrow spine issues I am about to look at.
Per the Easton charts, for 70 lbs, one cam, 2219 shafts with 100 grain heads would be within acceptable spine at 29" length, the charts do not show 2219 as appropriate at 31" length, so if you are going to stick with those arrows, you may want to trim them down some to get in the optimal spine range. Personally, those arrows are WAY heavier than I would want to use for most game. Your total grain weight is going to be over 500 grains. If you are going to shoot BIG game, like bears, moose, etc., that might be OK, but your arrow drop beyond 20 yards will be significant, so knowing the distance to your target will be much more critical with these heavy arrows. I shoot 70 lb 2 cam bows, and use Easton carbon arrows at 400 spine, at about 375 grains total weight.
Back to your tuning issue. The extra long arrows could be one part of it, so I would at least trim those down. You mentioned a "fork rest", I assume that is a TM Hunter style - You also mention 5" fletchings. Whether helical or offset straight, it will be virtually impossible to get good vane clearance with 5" fletching and a fork type arrow rest. There's just too much vane surface to thread between the forks. You might be able to tune the spring tension down a good bit to minimize the influence, but getting perfect clearance will be tough.
My honest, personal advice is to get some shorter, lighter arrows (maybe in the 2314 or 2413 aluminums, cut an inch longer than the front of your rest at full draw) with shorter - say 3 to 4 inch straight vanes. Your front of center balance with 100 grain heads will be much better, your arrow speed will be much higher, and the arrow diameter will be a tad larger, but with smaller vanes to help get better clearance. These arrows and a good drop away or full-containment rest would really help eliminate arrow or rest induced tuning problems. You'd still have to make sure the cables aren't touching, and you're not torquing the bow, of course, but these would be my suggested first steps.
I am sure there are plenty of folks that would argue that a heavy arrow is always better, but I think yours are too heavy & too long, and likely part of the problem.
Some more things to check...
On the peep sight, you'd probably be best off to start with a tube-aligned peep sight, so you can be assured it will be in line. I never could get any "stock" strings to keep a peep straight (& Martin has some very good stock strings.)
If the 6" drift stays consistent, even after you move your sight pin, it makes me think there must be a fairly consistent influence, perhaps it is the rest prongs steering those 5 inch fletchings. Are you shooting with the cock fletch down (threading between the prongs) or up, so the other two fletches are straddling the prongs ? Whichever it is, try it the other way and see what it does. If the answer is that you're shooting with cock-fletch out, traditional style, but shooting a prong style rest, there is no question you cannot get vane clearance.
You can use lipstick, dry erase pen, foot powder, or other such non-drying indicators on the prongs, then shoot and see how much vane contact you are getting. I suspect it might be a lot, especially if you're shooting cock fletch out - I think Stalkers are glue on nocks, so I guess it's not possible for you to easily move the nocks around....
I think I have some old 2413 shafts with uni-nocks down in my basement - I'm not sure how many, & some may need re-fletching, but I'd be glad to send you some to try out and see if they make a difference - they're just collecting dust since I went to carbon arrows a few years back. No charge - just returning the favor of those that helped me get started.
If you're interested, just let me know here or by Private Message.
Good luck & let us know how it works out....
If you change rests and still find the arrows drifting consistently, I would go back to my earlier statement about the arrow length. I haven't calculated it, but I would think the front-of center balance is probably not where it needs to be on 2219 aluminum arrows 31" long with 100 grain points. Compound that with the potential influence of long vanes steering off a rest, and I can see how it would be very difficult to make them hit where you aim.
This is a pretty easy thing to test. Just pull the inserts from 1 or 2 arrows, cut off 2-3 inches (I'd go with 3 or more) using a copper tubing cutter, and see if it makes any difference.
Oh, & I did cringe at the springy rest - if you are using plastic vanes, at least. If you are using feathers, it might not be as bad as the fork prong rest.
Good luck, & just let me know if you decide you want to try some of these lighter arrows. I think they'd be long enough for you to use.