Brents IT Blog

Random thoughts by an IT GOAT


Test Test and Test again.

Maybe this is part two of why telecom companies suck, but i figured i would name it with my point in mind.

So monday rolls around, i show up early with high hopes the phone number ports will go well.  I call into the conference at 7:55am, my phone guys log in 2 minutes later and at 1 minute till 8am, the telecom vendor rolls in.  What follows next can only be described as a near worse case scenario.

8am: Telecom company guy waiting for port company.
8:30am: Telecom company guy now frantically looking for port company.
8:45am: Telecom guy offers to continue at 5pm, i explain as calmly as possible, that we have already changed out phone systems and have no way to handle the calls.  Telecom guy pauses then says he will be right back.
9:15am: Telecom company guy says port guys should be on shortly.
9:20am: Port guys join call..  cannot find the information they need.
9:30am: Port guys still confused as to what they are doing, telecom guy explains over phone, discusses emailing information.
9:45am: Port has been completed, port guys take off, telecom guy says everything looks good, port was successful.
9:50am: All ported numbers say numbers not in service.  ( at this point, i wonder what i was thinking when i decided to go ahead with the phone system swap after being promised that ports would happen at 8am )
10am: Problem fixed on telecom side.  Numbers updated a second time, now the update has been sent out, but will be delayed since previous update was sent in recent past.
10:05am: Site managers calling saying patients say numbers they call are disconnected ( sigh ) ( boss stares at me )
10:15am: Numbers start ringing in( Phone calls off the chart, call center managers head starts to throb )
10:20am: Site managers call in saying numbers are ringning busy.  Confirmed via cell phone.  Telecom guy scratches head.
10:25am: I point out that there is no way this could be happening since there are nearly 100 lines.  We load up trunk tool and discover the other PRI circuits are not rolling over.
10:30am: Telecom guy says roll over fixed.  Phone calls really pour in.  ( call managers head begins to bubble. )
10:35am: All calls are inauditable, Yelling over conference phone at telecom guy is unsuccessful.  Calls my cell phone. ( call managers head explodes ) ( boss stares at me with a do something look )
10:40am: Problem identified, not enough bandwidth on fiber circuit.
10:50am: Bandwidth issue resolved.  ( boss states displeasure with telecom company, questions why we went with them )
10:55am: Calls not routing properly, call vendor ask why, vendor fixes issue, dental folks not recieving any calls.  They also fixed the dental call. ( boss questions why we went with VAR )
11am: Boss lets me know the music on hold for call center is horrible.  Determined that shoretel doesnt have any other good music.  (sigh).  ( boss happy now )

So lesson learned.  Stress test telecom companies stuff after they install it.  Then check it again before you make it live.  We stressed tested the PRI's before but someone at the telco blew them away the week after and rebuilt them.  Apparently they messed up the rebuild and we never verified.

So after this disaster, we decided to yet again bring up more sites.  This time, it would be the three smallest sites.  Unfortunately, as we would come to find out, they were not ready to port the numbers( yet again ).  I can say i havent had single port ever go smoothly.  It doesnt seem to matter how much you prepare for it, someone will screw something up, there are just too many folks with their hands in the pot.

Luckily the issuses were resolved somewhat quickly, though we still have a random voice quality issue that i am working on now.  Should be interesting to see the results of all but two of our sites ringing into the call center tomorrow.