Troubleshooting Annoying Problems

You would think that when purchasing pre-made, or manufactured, cable they are built properly. Millions of XLR mic cables are made and you would think the process would be flawless. When it comes to troubleshooting annoying issues, trust your skills and your instinct. You may be surprised what you find.

My annoying problem was a noise complaint within our road system for play-by-play. This system was built 5 years ago, but only this Spring did it become a problem, but only locally in the booth. The talent and producer complained about a noise that came and went, got louder, then soft, and did not seem to go away when equipment was connected, moved, or changed out. Granted, there was always a low level background noise, but never one that created any nuisance. After Spring Training was done, I had the gear dropped off for further evaluation and troubleshooting.

My first approach to troubleshooting is to recreate the issue if I was not there to witness first hand what was going on. Maybe how I set something up a bit differently than the end user, and we all know when something goes out in the field, the users do not always do what they were instructed, and once out the door, controlled environment can and will change, causing the field person to make changes. In this case, I setup just as I would if I was “out there”. As I had a few discussion with the producer already, I was initially looking, and listening, for a ground loop type noise and issue. In addition I was looking for a bad ground connection in general as this noise grew louder at times. It also changed at different venues. I hear noise. I also knew it was in the monitoring side of life as this noise never made it to air.

After a few minutes of wiggling and checking if all connections were good, the background noise we grew accustomed to was there, but louder, so I understood their complaint. As I checked all cabling for the monitoring or local, return, and talkback audio, I was not able to isolate it. At this stage it was time to methodically check each and every connection for each talent. Starting with channel 1 on the mixer I stepped through and compared listening for any nuance between positions. I also expanded my search to outside the monitoring. I soon discovered that if I unplugged a mic cable the noise went away. Interesting to say the least as I mentioned this noise never made it to air. I moved the cable to another input, and discovered the noise continued. As I also know the equipment gets jostled around a bit while traveling I decided to check for any loose mixer components. My method: Treat it like it is used in the wild. So, I gave it a solid, but not destructive pound with my fist. With said mic cable plugged in, the noise got loud, then dropped again. I repeated without the mic cable in. Silent. Mixer good.

I have not isolated the issue to a mic cable. Remember, this noise was only in the talent and producer headphones and not on air, yet the mic cable was definitely an issue. I swapped with a known good cable, and no noise. Always use a known good at the point of localizing the potential culprit. In some cases use more than one known good to test and be sure. Out comes the cable tester to verify my findings.

Here you see an “X” pattern on the tester. Bad Boy This is definitely not normal, and this is a pre-made, or manufactured cable. This mic cable was made this way. Look closely at the pattern and notice that this cable was made unbalanced. This was not a “failed” cable as I stretched and twisted and kinked the cable throughout the length to induce an intermittent. It never wavered.

Now this is the cable test of the known good cable which I eventually used as a replacement. Good Boy As you can see here, there is a nice one-to-one pin match between the two ends.

After the mic cable was replaced, I tested the whole system again. This, my friends, is a very important step. Though at this time I found the issue, I want to make sure there are no others. In this case all was good, so I put everything back together. Once secured, I did one last check.

Why the noise from this cable? A good question. As the picture showed, this cable was made unbalanced, and since the low, pin 3 side, was connected to ground, pin 1, all would work just with reduced level. For a mic input on a mixer that is electronically balanced, with a transformer, the common mode rejection was thrown out the door. In addition, the cable itself was now susceptible to noise and could act as an antenna, allowing this noise into the system. As headphones are wired unbalanced this noise can carry through to the headphones. Since the output of the mixer is balanced and wired to the balanced input of the codec, the noise is rejected as it appears on both the high and low signal leads which are out of phase feeds, and no noise on the air.

All this was quickly fixed by just using common troubleshooting practices. Throw a little instinct in there, like the rap with the fist, and we move on to repair or replace. This setup is currently in use this weekend and no annoying noise. Talent and producer are happy. Hope this information helps, and don’t rule out pre-made cables, they could bite you later.

Cheers!

Categories: Equipment Tags: ,

GatesAir FMXi 4g Metadata

Metadata. Title, Artist, Album information. Artist Experience. PSD. Data. All data. Different names for data. What can be so difficult about that? Everything. Format, type, destination, were it comes from. The FMXi says you don’t have to do much to get it working. Well, I beg to differ.

I will start with the easiest form of data used in HD, program data. This is known to some, all, few, as PSD, Program Service Data. Basically it is Title, Artist, and Album information. Nothing special, though different devices accept different formats. Today, the formatting is pretty much standard. According to GatesAir one can send PSD data to any of the three NICs on the FMXi. The network interfaces are labeled as Management, E2X, and PSD/IMP. Each is separate and the device is its own switch and router. Easy enough. To keep things simple and create a standard I chose to send the data to the PSD/IMP interface. Gosh, PSD is in its name, so why not use it.

First be aware that none of the three network interfaces can live on the same subnet. I tried to configure the PSD/IMP NIC to reside on the same subnet as the E2X. Before this device I had my Importer and Exporter living on the same network. Flat network. Simple. To make this installation a bit easier, our IT made a subnet which is basically the metadata subnet. Arctic Palm and JumpGate reside on this network and worked prior to the advent of the FMXi, so logic dictates to send this data to an interface on this same subnet. Guess what. Did not work. Some digging to do and from the Arctic Palm computer I can ping this new port. Yeah. Double checked port assignments. HD1 PSD data default is 11000. Easy enough, we are using the defaults. Nothing. Hold up. What?

Shoot, let’s try to send it the data to the Management NIC. By the way, this is also known as the WAN interface within the FMXi, so it creates a bit of confusion. I call it the house network, or management port. I know the interface works for the Web GUI or how else would I be able to configure the device? Send data there, I say. Nope. Not. Or as I like to say, Nicht, Nein. OK, remember my last post about routing tables and ARP? Let’s check them out! And, of course, all checks out. Arctic Palm and the JumpGate IP addresses show up in the table and references the proper subnet. Routing? Should not be an issue. Devices and interfaces configured to be on same subnet. Devices show in ARP table.

Let’s try the E2X interface. OK. A simple change again to the devices delivering the data. PSD shows! As my daughter likes to say, and I guess many teens these day do, wait, what? The routing table doesn’t show any “new” gateway. IP resides on same network. It works. Hold on a moment. The PSD/IMP interface is configured to be on same subnet as the devices delivering the data. Said devices show in ARP table. Not data. E2X interface works? Same with the management interface. I now conclude the claim that PSD can be sent to any of the three ports is false. Unless there is some firewall thing talking place on our network, there should not be an issue here. Side note, our network is so complex that only 1 person in our building knows at least part of it, but will not share with the rest of us. I prefer to keep things simple.

Time to move onto Artist Experience. Believe it or not, we got this one working before the PSD! At the start we did not get the data to flow properly, but after a little poking around we tried something that is not suggested at in the FMXi manual. Though the JumpGate which aggregates this information is on the “metadata” subnet and has access to the outside world to retrieve the artwork, etc, as we found out with PSD it did not get to the FMXi. This one was a bit different, and since we knew it had something to do with the outside world, it was back to the routing table. The decision was made to add a “catch all” default route. In the table it was time to add a destination address of 0.0.0.0, a netmask of 0.0.0.0, the gateway to the outside world which happens to be on the management interface and subnet. Once this was done, Artist Experience was on the radio. Simple fix, but not the most intuitive.

BTW, based on the FMXi interface configuration, the unit will define a “system” route based on IP configured. If you have the management interface configured for 10.10.10.10, the netmask will be 255.255.0.0. Or if you configured it static and chose a mask of 255.255.255.0 to keep it manageable, then that would show. The gateway for all these would be 0.0.0.0. By doing our “fix” we created a default type route which seemed to make the unit happy with Artist Experience. Maybe if more digging is done the route table could be manipulated such that PSD could be used on any interface. The issue is the manual does not say that. It does not suggest that. We make the notes so we know how to put the next one in.

I suggest if/when you install the FMXi 4g you brush up on the networking skills. If you have a simple network, the better. Unfortunately you cannot configure the three interfaces to live on the same subnet, so if you do keep a simple network, you will not have to configure all three, but you will want to configure the management interface as that is your Web GUI, and you will need your E2X interface configured to ship that stream to your Exgine. I do like the built in diversity delay and it seems to be holding, so that is nice. Good luck. I hope this information helps you install and troubleshoot the FMXi. Maybe in some firmware update GatesAir will include simple diagnostics, like Ping, to help verify network settings and routes.

Cheers!

Categories: Equipment Tags: ,

GatesAir FMXi 4g Importer/Exporter

Corporate mandate: Install this on your station. Reason: Old Importer runs Windows XP, security reasons.

I accept the reason, but I do not understand the mandate for a specific manufacturer’s equipment, especially when my transmitter is a different manufacturer. Making a GatesAir FMXi 4g work with a Nautel NV20. Well, I do, but money should not always be the answer to everything.

This can be done. What I did discover is quite interesting, and is not a very good way to determine proper operation. To make this whole picture come together imagine your choices of installation location for a combination Importer/Exporter. If you put it at the transmitter site, the Exporter sits on its own subnet with the Exgine. Now you need to get your audio for your HD2s and 3s to the site, in addition to metadata. If you install the combination at the studios, then you have your Exporter and Exgine on different subnets, or you are forced into VLAN fun. I have been running Exporters from the studio site successfully for years, so it can be done, but the FMXi gave me quite a start.

Per the manual, I configured the management NIC and the E2X NIC on the FMXi. I saved the PSD/IMP NIC as the last thing to worry about. When configuring the device GatesAir wants you to configure a routing table so traffic knows where to go. E2X is a simple UDP stream, so this should be simple enough. No HD. Head scratch. Example: E2X NIC given the 192.168.10.10 address. Exgine is on 192.168.20.X network, so the routing table is configured to send data to the 20.x network, use 192.168.10.1 gateway. Simple enough. No work. Discussion time #1.

Take the unit to the transmitter site. Basic troubleshooting principles. Drop it on the same subnet. After a quick configuration still nothing. Poked at the Exgine and verify ports are correct. Sheesh, haven’t changed those in so long, but verification away. After about 10 minutes of this, I noticed the FMXi was now happy, Exgine was detected. So, it can talk with the Nautel Exciter and Exgine. Cool. Back to the studios to contemplate why. Discussion #2.

Discussion #2 with Gates revealed a very important, yet minor detail about the Exgine being “detected”. I was curious about how a UDP connection got confirmation that “presence”, as Gates calls it, is determined. The surprising answer was the ARP table. The FMXi creates an ARP table just like any computer or switch. If the Exgine IP address is in the ARP table, then the Exgine is considered present. Guess what? If you are on a different subnet, you will NEVER get that “presence” indication, and your FMXi will report visually that the Exgine is not present.

As I had the FMXi working, as in creating an E2X stream at the site on the local subnet, I knew I could get this to work from the studios, so back to configuration and poking. Again, I made sure the routing table had the route required to get the E2X stream to the site. This time, I kept refreshing the ARP table until I saw with my own two eyes that the gateway configured showed up. Bam, HD was on. Here’s the kicker. The FMXi still reported that the Exgine was not present. So, now I have a box that is working, but reports that it is not. Here’s the proof:

Exgine Fault but HD Carriers on Transmitter

Apparently GatesAir will have to come up with a “better” way to inform the end user that the E2X stream is being sent to where it is designated to go. In all this, I also discovered that the FMXi has no built in diagnostics such as ping or trace route. If I was able to ping from this device I would have confirmation that my route was correct and that the ARP table had correct entries without having to go through these extra hoops hoping to see the HD carriers and hear audio.

There you have it. Intermingled manufacturer devices working together. The Nautel HD Multicast+ was not this difficult to setup. I will address metadata in my next post, as another interesting discovery was made regarding PSD data. Until then!

Cheers!

Categories: Equipment Tags: , ,

Happy Lunar New Year!

February 11, 2021 Leave a comment

Wishing those a happy and prosperous new year. This is the Year of the OX!!!

(Doh! I put goat by mistake in original post.)

Let’s hope the attack on Covid-19 continues to progress and we can get things back to semi-normal sooner than later.

Cheers!

Categories: Uncategorized

HD Radio = Headache

January 27, 2021 Comments off

What is up with HD Radio? Why is it so complex when normal networking protocols have been in place for years? Ever asked yourself these questions? The end user does care how it is received, but if you have to install and configure such a beast you could go bald pulling your hair out. (Assuming you started with hair.)

Since the inception, we have managed to run the E2X, Exporter to Exgine, stream over our MPLS network. We have all the equipment at the studio end for simplicity. More on this in a bit. My 3 stations were humming along without issue until it was time to replace old, aging, or even failed equipment. Now all of a sudden where this stuff is located becomes an issue.

Manufacturers tend to do things differently. This is to be expected. Each radio facility is expected to do things differently. Again, expected. My issue is that those who design and provide this stuff think that radio facilities do everything the same. Guess what, folks. Each facility is build with some basics “standard”, but customized to fit the need of the format, personality, and mentality of those that build it. For my installation I look toward simplicity. I like to keep things simple, so why cannot I install my Exporter or now combo box at my studio location? I’ve done so in the past, but now your new box won’t work this way? Does it not make more sense to install this at the studio especially if it is a combination Importer & Exporter?

Let’s evaluate this. If a combination unit is installed at the studio, then I can have my metadata, my other HD channel audio, and my control locally with a SINGLE data stream to the transmitter site. If this unit is installed at the transmitter site, then I must send a data stream with metadata AND multiple audio streams to the transmitter site. Bandwidth becomes a real issue, especially since we, here, are a full IP based STL distribution system. It is just more convenient and efficient if this is located locally at the studio end. All I need to send to the transmitter is a single data stream for HD and my main audio.

Right now Nautel makes this happen just fine even with their new HDMC+. As I write I have discovered that Gates requires the E2X to be on its own subnet. Nautel I just drop the data on the network at it goes. To make the Gates work on one station an extra box is required, called the IPLink. This encapsulates the E2X on the pipe that connects to the site. Why do I need another box? Save money they say. How? You have to buy another box or redesign your whole audio chain. Seems crazy.

Whatever happened to standard networking protocols? Why does HD data have to be so special? How different does it have to really be? We can squeeze Zoom meetings with video and audio on fairly conservative network, but we cannot send a simple, supposedly, UDP data stream from point A to point B on a different subnet? A private one at that?

If we did everything the same, then we may as well have automatons do our work. Thanks radio for spending so much time eliminating the people that make the medium unique.

Cheers!

Categories: Equipment Tags: ,

WideOrbit and Dante

January 14, 2021 Comments off

This year started off well. Learning something new everyday and applying what has already been packed away in my little brain cell.
Though not 100% resolved, I discovered an interesting thing with WideOrbit in the last couple weeks. After a certain version of their Audio Server, you must specify within WO the sample rate of your recordings, explicitly. I found this out after another staff updated the audio server on a workstation that has been acting a bit odd. After this installation, the workstation would not record at all. Only thing that changed was the audio server. Now what?

Scouring the logs, from audio server to automation and back there was a missing piece of the puzzle. Tech support was a bit baffled by this, too. Finally a key phrase popped up in the automation log; an error. Sample rate error. Not very specific, but interesting enough to test. Dante is native 48kHz sample, and our house/facility is clocked at 48k. We do background recordings throughout the morning in WO using workflows. So, we decided to specify in the workflow the sample rate. It worked. We, this is me and WO support, decided to do some other quick record tests to put it through its paces. Each worked when the sample rate was explicitly set to 48k.

We took the test further and we got on a workstation with an older version of the audio server. No issues recording by touching nothing. The tech was floored. A new “feature”. Definitely a difference in the newer audio server. Needless to say, all workflows were updated and recordings have been just fine.

A radio station really does not need to clock at 48k, and 44.1k is pretty much standard. When the facilities were built, I made the decision to change the system to 48k to adapt to Dante AoIP. Dante runs at 48k by default. One main reason for this is to be compatible with AES67. If you wish to interface with an AES67 device, you must run 48k. Another item of note when running Dante is you cannot change the sample rate of a device to 44.1k and have it communicate with other Dante devices set at 48k. All Dante devices which need to communicate on the network must have the same rate. I confirmed this with Audinate support.

Make sure if you plan to install an AoIP network using Dante that you choose your house sample rate carefully. I think flexibility, so 48k was the obvious choice. This time it caught us, but at least it was an easy fix. Consider this when working with other AoIP solutions. If you wish to be compatible with AES67 in its basic form, your sample rate will need to be 48k.

Cheers!

Update: 27-01-2021.

So we come to discover that the version of WO we run is not fully compatible with the “new audio server” (NAS). Had to drop back and punt. We reinstalled the legacy audio server for this workstation. I mention to the tech that I should test a more recent NAS install on another workstation and see if I can break it. Strongly suggested I should let it be. Nice.

Categories: Equipment

Happy New Year!

December 31, 2020 Comments off

Greetings my fellow techies. I hope each and everyone is safe and healthy. Let’s hope 2021 brings us closer to normalcy as we start getting vaccinated.

Cheers!

Categories: Equipment

Happy Friday

December 4, 2020 Comments off

Awaiting HVAC tech. How about a picture?

Categories: Equipment

Quick Check In

December 3, 2020 Comments off

Why we need HVAC all year round? This would be the sunset tax. Tomorrow I meet the tech for an ailing HVAC unit.

Categories: Equipment

Logs

August 10, 2020 2 comments

Now that is a boring title, eh? Why logs? Why write about logs? No, there will be no examples of logs, but if you do any sort of troubleshooting it is good to read them logs. I know many of us do not really read these things to deeply, but over the years I have solved many issues just by reading the logs.

Today is a good example of the “win” when it comes to logs. The traffic department had shown discrepancies and missed spots during a portion of the day. Everyone gets up in arms when spots are missed. I also notice that fingers get pointed. Long story short, I first checked the logs to verify the report of a system restart. Found that. The next confusion was why did the spots not run. Using the logs I verified contact closures being recorded proving the system was working just fine. Back to the logs and I found that someone left the system in manual mode. No more finger pointing now, eh?

This sound simple, but it can be tedious work. As you know details are always lacking when people report a problem. Doing the detective work means really reading and finding the clues within the logs. Today was WideOrbit. If you ever looked at their radio automation log you know how much detail (and fluff) is in there. With a little patience and time you learn what is important to you and what is not. From there you use what you find to get to the solution, and in my case it was not technical issue. Someone decided to restart the system and forgot to change to automatic mode.

I comb through many different logs based on what I am working on. SAS, Dante, WideOrbit, NexGen, Nautel, and the list goes on. All have logs, and many are very useful. Granted some are not as useful as others, but for the most part they are good to getting the job done. Take the time to read the logs when you troubleshoot. It may take time to decipher, but the more you do it, the easier it becomes. After which you can then tell the manufacturer they have an issue, provide a copy of the log as supporting evidence, and see how fast they respond to your support request.

Cheers!

%d bloggers like this: