About four months ago, I made the decision to extend my lab, Tsoorad.net, out to the larger world of the internet. I have several reasons for wanting to do this; most of them selfish, but a surprising number of them are for client testing purposes. I need/want to implement a full SBC for a SIP trunk, which leads to needing a SIP trunk, et cetera. So for this exercise I need more than one (1) static IP. I could probably figure out a way to futz with ports per service and get something worked up, but I want to be able to duplicate customer scenarios. Internally I have plenty of network. Externally, I need a static IP configuration. And with needing at least three IP externally, the logical choice (see how I justified all that?) a /29 seemed to be a good fit.
Enter Frontier FIOS
I have been a fan of FIOS since the halcyon days of Verizon replacing my DSL with fiber to my house. Awesome. None of the crap that you put up with if you use cable. I don’t share my bandwidth with the rest of the neighborhood. Nice and consistent on the speeds up and down. And the 15/5 package is more than sufficient for my 2 person internal domain and my lab domain.
But, it turns out that FIOS has this ONT thing (Optical Network Thingy)(actually, Optical Network Terminal) mounted to the side of the house. Which leads us to the The Problem.
On the Frontier side, the sales folks sold me. I know, I should have been smarter. I should have known. Sales types around the world have a mantra: “yes yes yes, please sign here” and they are compensated on their ability to get the signature. I get it. But I hate misinformation.
Knowing that I was going to need at least three of the /29 workable, asked the obvious (and brilliant) question: “Will the Frontier supplied router be able to handle a /29 or will I need my own firewall/router?” I should have heard the alarm bells go off when I had to explain what a /29 was and why I was asking the firewall/router question. But, I was assured, by someone who assured me that they worked on the business provisioning side of things, that the router supplied on the install would handle the requisite services.
When the Frontier install tech showed up onsite, he expressed his doubts. But, armed with the knowledge (however faulty) of the sales professional, we moved forward. The actual connection part was easy. When I first moved in, I had the technician run Cat5 from the ONT to the internal drop. There are pluses and minuses to this approach. But seeing how I could care less about the TV service (I use a popular satellite vendor), using an Ethernet drop versus coax was a no brainer. As it turns out, a necessary one as we shall see.
Firstly, if you have coax to your local demarc, then you will need to use the Frontier-supplied router/modem. The Frontier-supplied unit, an Actiontec MI424WR, certainly LOOKED like it should handle things. In the network definition section in the setup, there is a place to put in IP’s, and it would take the full definition complete with mask and gateway. But, we could not get more than one address working. At one point we had a second address working, but it abruptly quit after a few hours. For reasons we shall see, this was caused by the ONT and in the end, had the Frontier Tier 1 non-help desk been more informed and less script-driven, I might still be using the ActionTec. You see, the problem, while it appeared with the original Frontier-supplied router, might not definitely indicate that the ActionTec won’t work. However (on the flip side) I did have a longish chat with an Actiontec “engineer” who blithely told me that their router would only do 1:1 static NATs, and not dynamic 1:M NAT. I am also not sure if the Actiontec would be able to do 1:1 AND 1:M NAT at the same time. I suppose that particular test will be done at some point in the future when I am snowed in or can’t sleep at 0230.
Next, having proved over and over that we could get all 5 IPs working off a switch on five different pieces of gear, and all at the same time, we started looking at how the network was delivered from the Frontier side of things.
<Cue creepy techno-vibe music from CSI with lots of tracert, pathping, and telnetting with multiple interfaces on different machines for configuring the various firewalls and other appliances>
Back to the Frontier Tier 1 non-help folks who (reading from their script) had me retest everything again, and then proclaimed that I had a DNS issue. <sigh>
Over and over this went. Then it turns out that the network technician who was advising the Tier 1 person was reading from the standard, world-wide network engineer script that goes something like this: “Everything is good here. You have a problem.” And the idiot user (me in the case) is not allowed to speak directly to the actual network technician – so you have this wonderfully frustrating back ‘n forth between the idiot (me), the non-support Tier 1 person, and the network technician who blindly insists that all of their gear is provisioned correctly. Because I was asking for support for something that was not common to their network, this entire effort failed because I was off script.
This went on for several months. In that time I tried, in order of implementation, the following firewalls: Cisco PIX 515e, Cisco ASA 5520, Cisco ASA 5505. Nuttin’! All this time, each time I called for some more non-help to be told how everything was peachy on their end, but gees, maybe you could clear your ARP tables please, I was told how the issue had to be on my gear being misconfigured. Now, I am not in any fashion a Cisco ASA expert. But I have some knowledge. So when it came to about 1/2 way through the 5520 efforts and for ALL of the 5505 efforts, I trotted out the hired guns. Namely a completely certified (he is really out there) Cisco dood – who lives and breathes Cisco security as his primary job.
As a side note, this guy was awesome. But clearly, in the end, he was clearly puzzled as to why what works for the rest of the civilized world refused to work for the TsoorRad environment. And he did this several times. We had nothing wrong. My network is solid.
The Fix Begins
At about this time, I am starting to doubt my sanity. Stupid thing should be working but it is not. I think that I have eliminated any possible error on my side – Yet Frontier network techs still insisted that there was nothing wrong on their end. However, if you read some of the material to be found online which I magnanimously have provided here:
…you can see that I am NOT the only one. However, in defense of fairness, in some of the research the firewalls were called into question. So, I will try one last time. I purchased a NetGear FVS318G. Same results as before. I then discovered a piece of the Frontier website that allowed me to find email addresses for executives in my area. So I fired up my email and sent a politely worded notice of my dissatisfaction. Several. But the first one got serious attention. So the buck does stop somewhere.
So now I have Frontier calling me, managers, technicians, and technician supervisors all have been told to fix this. About time. But guess what? The network team on their side is still insisting that their configuration is perfect. Here is a little blurb of the back-n-forth:
“…the ONT cannot (empirically) deliver the entire /29 all at one time to one device such as a firewall. This is not acceptable.
It would appear that something in the ONT is not able to send the entire /29 block to one device. I have now used 4 different firewalls, ranging from bottom end business class (netgear fvs318g) to industry standard business class firewalls (PIX 515E, ASA 5520, ASA 5505) and they all do the same thing – specifically, the devices work just fine on one IP address, but the ONT fails to deliver the entire block to the single device.
I think that the ball is back in Frontier’s court. Something must be able to be done to the ONT to bridge my entire /29 to the Ethernet so that what I paid for, namely business class service, is actually delivered to me. Bridge the ONT, tell the ONT to handle multiple IP’s, upgrade the firmware on the ONT to do this, SOMETHING…”
The Fix is IN
As it turns out, the ONT not being able to deliver all the addresses was the issue all along. While I cannot get direct information as what EXACTLY was done, my service now works with a complete /29. Somebody up in Frontier-land did something to either the ONT (several folks did several things, all very hush-hush) and then an upper-level regional engineer did something to the upstream router (from my service perspective) and magically the NetGear FVS318 is cranking on all five cylinders. I cannot verify the regional engineer thing, it was only mentioned once, and then when I enquired about it again the subject was changed very quickly.
Here is what Frontier told me was the issue:
“Our Gateway Router sends out an ARP request every 6 hours or so to see if the same MAC address is still using an IP address. It sends it out on 0.0.0.0 and if it does not get a reply, it shuts down that IP addresses' ability to get online. The reason this happens is because many firewalls identify an incoming request on 0.0.0.0 as an attack. So the Firewall sees it and then discards the packet. The discarded packet never gets back to the Gateway Router, so the Gateway Router in essence says: don't let this IP address get online, I cannot verify the Mac Address.
At the time you were experiencing this problem, we were introducing a new optical network terminal (ONT) into our network; Calix. It operates much different that the way the Alcatel Lucent system operated. We plan on exclusively using the Calix ONTs for small business customers where it has been added, which is about 95% complete in the Oregon FiOS markets.”
- Remember the Actiontec that did two addresses but then quit? IF the ONT is configured correctly, maybe, just maybe the Actiontec CAN handle a /29 like the original sales person stated?
- The ASA is a multi-talented device. Dang did I learn a lot. But in the end, not able to work for me. My buck-twenty NetGear POS is working just fine. Complete with SIP trunk.
- Frontier has assured me that the process has been changed as any business-class (fixed IP) service will require the new Calix ONT (mentioned above).
- Sales types are a necessary evil. Business doesn’t work without them. Caveat Emptor. Be educated for yourself. I don’t know if I could have done anything different, but I will be thinking about it for future actions.
- When I got ahold of the correct layer of responsibility inside of Frontier, things went very smoothly, and I was treated much differently. Like maybe I knew just a little of what I said.
- Network technicians need to learn to look a tad past the CLI on their screen and stop assuming that “everything” is correct on their side of things.
In the end, I am happy, but I sure wish this had not taken so long to resolve.