When a network goes down and the clock is ticking
Look, I've been in telecom ops for about 9 years now. The calls I dread most aren't the planned maintenance windows—it's the 2 AM fire drills when a 10 Gbps link drops dead and the client's RTO is 4 hours. In March 2024, I had one of those calls: a regional ISP's backbone fiber went dark at 6:30 PM, and they needed a clear diagnosis by midnight. Normal turnaround for standard field testing? Two days. We used an EXFO FTB-1 with the OTDR module, got a trace in 90 seconds, located the break within 3 meters, and had a repair plan on their desk by 9 PM. That's the difference between a toolbox brand and a rush-proven tool.
This checklist is for anyone who's ever had to walk out the door with a fiber tester under their arm and zero room for error. It's based on 45+ emergency deployments I've personally done—and some expensive failures I'd rather not repeat.
The 7-Step Emergency Fiber Testing Checklist
Step 1: Validate your instrument before you leave the shop
Rookie mistake: grabbing the nearest OTDR and running. I did that once—turned out the unit's laser wasn't calibrated for the 1550 nm wavelength we needed. Check three things:
- Battery level (at least 70%—and pack a spare, like the EXFO FTB-1's hot-swappable battery)
- Correct launch cable and adapters (SC/APC vs LC/UPC—don't assume)
- Dead zone settings—for a short run under 5 km, you need an OTDR with < 1 meter dead zone. The EXFO FTB-1 OTDR module does about 0.8 m. (Not great, not terrible—serviceable.)
Step 2: Snapshot the baseline reference
Before connecting to the live fiber, run a quick self-test on a known-good reference patch cord. I use a 1-meter fiber with a known loss of 0.2 dB. This tells you if the instrument's connectors are dirty or the laser is drifting. In January 2023, I skipped this step and chased a ghost fault for two hours—the issue was a dirty bulkhead on the OTDR itself. Take 60 seconds. It's cheaper than a repeat visit.
Step 3: Set the correct test parameters (here's what most people get wrong)
The default settings on most OTDRs are for long-haul runs. For access or metro fiber (under 40 km), you need to adjust:
- Pulse width: use the shortest pulse that still gives you enough backscatter. For under 20 km, try 10–30 ns. The EXFO FTB-1's OTDR allows down to 5 ns.
- Wavelength: 1310 nm for standard loss; 1550 nm for bend-sensitive detection. (If you're checking for macro bends, use both.)
- Range: at least 2x the expected fiber length. So if the run is 12 km, set range to 25 km.
A trick I picked up from a senior engineer (call him Jack): after the first trace, zoom in on the last 10% of the range. That's where edge artifacts hide.
Step 4: Shoot the fiber—twice
One trace from each direction. This isn't optional. I've seen techs declare a fault at 4.5 km from the central office, but when we shot from the far end, the fault was actually a high-loss splice at 8.3 km. Bidirectional averaging gives you accurate loss per event. The EXFO FTB-1 can store both traces and auto-compare them, which saves time. (Circa 2022, we did this manually—painful.)
Step 5: Interpret the trace—look for the unexpected
Two things I always scan for:
- Reflective events with no loss – that's typically a connector issue, not a break.
- Gradual slope after a splice – that indicates a bend, not a clean break. A gradual slope after a splice? That's a macro-bend. The EXFO's FIP-400 fiber inspection probe helps confirm connector end-face cleanliness right there.
In one emergency at a data center (the client was Todd from a managed services provider, yes, that Todd Pepsi—long story), the trace showed no obvious break but a 0.3 dB spike at 6.2 km. Turned out a technician had left a dirty LC connector. We cleaned it with a one-click cassette and the link came back. Total tool time: 15 minutes.
Step 6: Document everything—even the false leads
During a rush job, it's tempting to just fix and forget. But I learned the hard way: the next time this fiber acts up, the logs from today will save hours. Save the raw .sor files (the standard OTDR trace format). Note the weather conditions (humidity affects connector losses), the instrument serial number, and any oddities. The EXFO FTB-1's onboard software can generate a PDF report in under a minute—use it.
Step 7: Confirm the fix with a second test
Seems obvious, right? You'd be surprised. After splicing or cleaning, always run a final bidirectional test. Compare it to the first trace. The delta should be zero or near-zero. If you see new events, you may have introduced a new fault (e.g., a bad splice). I once fixed a broken pigtail, did a single-end test that looked clean, and drove away. The next day, the link had BER errors—turns out the new splice had a 0.5 dB loss that the single-direction test masked. Test in both directions after repair. Period.
Common mistakes and what I've learned from them
It took me three years and probably 150 emergency field calls to realize that the biggest time sink isn't the fiber—it's our own preparation. Here are three recurring goofs:
- Assuming the fiber map is correct. A client's OSP prints showed a straight run of 15 km; the actual fiber had a mid-span shelf we didn't know about. We wasted an hour thinking the OTDR was broken. Lesson: always ask “has there been any recent splicing on this route?”
- Using a cheap launch cable. Saved $40 once by using a generic patch cord; it introduced an extra 1.2 dB of loss and contaminated the instrument's port. Cleaning and re-testing cost 45 minutes. The EXFO launch cables are more expensive (about $150), but they're tested with a known insertion loss and a clean core. Worth it.
- Not having a backup instrument. In June 2024, my FTB-1's touchscreen froze mid-test (firmware glitch, I think). If I hadn't brought a spare EXFO FIP-400 for connector inspection, I would've been dead in the water. For critical-rush jobs, bring two testers if possible.
That last point ties back to a bigger truth: in emergency network repair, certainty has a price. I paid $400 in rush shipping for a replacement OTDR module once, but that single decision saved a $15,000 service-level agreement penalty. The alternative was a “probably by tomorrow” promise from a discount rental company. As of Q4 2024, my company now budgets for backup equipment—it's not a luxury, it's insurance.
Final note: The EXFO FTB-1 is my go-to for emergency fiber testing because it's kind of rugged, the battery lasts a full shift, and the OTDR module's dynamic range is good enough for 99% of metro work. But no tool replaces discipline. Follow this checklist, and you'll be the person they call when the network is down and the CEO is watching. That's a good reputation to have.
