WideOrbit Automation & EAS

When we made our move into new facilities, we had to change our automation/playback system from NexGen to WideOrbit. At first we thought that it may be a good change. Maybe even take care of or streamlining some of the processes. The plan was to continue with an IP based facility, so we needed to make sure there was not falling back to a ton of cabling and physical GPIO for this. Well, a year+ later and we continue to fight how to reliably and consistently run EAS tests from WideOrbit.

As we saw this was turning into and issue, we setup our Sage Endec to send a RWT on Sunday mornings while we worked out the details of how WideOrbit deals with them. This has turned out to be a good thing as various attempts and inconsistencies caused scheduled tests to be missed. We would make them up, or attempt to, as soon as we found out of such events, but some got through. This begged the question why? We stuck one of our guys on it. Week after week, months gone by, there was no rhyme or reason to why some worked and some didn’t. Unfortunately we had cutbacks and the person working on this was released. The void had to be filled and I took up the challenge.

I had, with a little work with NexGen support, the Endec and NexGen working well via Ethernet. Weekly tests were scheduled and they would run. Monthly tests or alerts came in, and NexGen would drop them into the log, fire them, and resume operation. This was 7+ years ago! I believe I even wrote about it in a previous blog. I began to dig in and find out what is so different here that we cannot do the same.

The WideOrbit documents show a log of information if I wired up GPIO, and they do have the EAS widget. I began questioning operators on how they used the system. What did they do? Who did what when it worked? Who did what when it didn’t? Yes, using the widget produced different results. I decided to dig deeper and pulled out the trusty Ethernet tap and placed that in front of the Endec. Data gathering time has begun. I learned a lot of what the Endec sends and what it wants to see. I sent notes to Sage support for clarification. I now know what the heartbeat is. I know what a query from the workstation is and the Endec response. Time to see what happens when they run their tests.

My packet captures were full of fun stuff. One forgets that the Endec, and all EAS units, communicate with the CAPS servers, so all that traffic I knew had to be ignored. Pretty cool though to see the chatter, but time to filter that out. I only cared about the talk between the workstation and the Endec. There could be much, but then I noticed the Endec sending a series of strings consistently timed. The heartbeat. The Endec sends a heartbeat to tell systems the unit is there waiting to play. Included in the heartbeat is the expected delay if a RWT is run. In our case it is 14.45s. I never knew that, so something new is learned. Then I see a packet sent from the workstation and the Endec immediately replied. I learned that this means the EAS widget on the workstation was definitely connected as this was a query to the Endec, “are you there?” The Endec sends a reply code, so all good there. Why are tests not being run?

After much digging, and observations, I still did not know why the inconsistency, so I reached out to our engineering community. With a couple of responses of positive results, I needed to chat. The main thing that came up and was consistent between the two engineers was the fact that a test run via the widget was dropped into the stack. From the information I received from our operators some would drop the test into the playlist/log in advance of the test, while a few would drop it into the stack. I took this information and decided to put it to the test. I went to each of 4 stations control rooms and asked, stack or playlist? The answers were mostly playlist insertion with a couple of stack if it was convenient. Each station we dropped the test into the stack. Each time it triggered. Part one of the mystery may be solved. Instructions went out to the air staffs’ to use the stack insertion.

All this fine if you have someone in the studio to run the tests. These days stations are unattended a good portion of a day to all day. How does a system like this deal with these situations. Turns out WideOrbit really does not have a system when using Ethernet or IP control. If you are wired for GPIO they have some provisions. So, I decided to experiment and see if I can at least fire a RWT using a workflow. I configured a workflow with the instructions to connect to the Endec’s IP address and port, and send the string to run a test. To make it testable, I put it on a hotkey so I can fire it at any time. The moment of truth, I click on the hotkey. I check out the Endec. Nothing. I do it a couple more times. Nothing. I create a telnet session from the workstation to the Endec and send the command. Works like a champ. I check the WideOrbit logs and they claim the workflow ran and sent the string, but the Endec did not respond nor did the packet get captured, but he telnet did.

At this time I am at a loss on how to make a completely working IP control of the Endec using WideOrbit. In today’s time and the proliferation of unattended stations one would think this would be of high importance. When asked, WideOrbit support responds with run the Endec in automatic and let it run. I hate to say it, but when you are in a top 20 market the last thing the PD, GM, and sales manager wants to hear is an EAS test stepping all over a commercial break or song. Sure it can be done with GPIO, but to me that is an 80’s, last resort solution. Leave a comment if you have any thoughts.


Categories: Equipment Tags: ,

Should I or Should I Not

May 23, 2019 1 comment

So, I’ve been a been absent on posting. I began to start wondering how helpful my blog posts are. Leave a comment if you think these are worth it.

Question: Do you want to see more posts?

Question: What subject matter, or should I post on what I am doing?

I also dropped social media due to the inundation of junk that was not worth weeding through. A shame that people cannot use tools properly and effectively. In any case, would a twitter feed by useful again?



Categories: Equipment

Happy Holidays!

December 19, 2018 Leave a comment

A long year is coming to a close. Here we were involved in the most complex and annoying facilities move ever. As I gather notes, I think I will post some highlights of the move. I am also going to take the age old advise and say, if you have nothing good to say about someone, do not say anything at all.

Have a Happy Holiday season and best wishes for the New Year!

Categories: Equipment

AoIP: Dante by Audinate

Love it. Simple. Just build your network properly.

We are what I would say is 3/4 the way through getting these facilities in a psuedo-complete state. The one thing that I think has been the most reliable is our audio network, Dante AoIP. Simple, efficient, and easy to maintain.  I need to do a count, but I am sure I have over 50 devices on this network/system.  I know I am not done.

The devices range for KDL modules in my SAS 32KD frame, to SAS Rio Bravos, and a multitude of Dante Virtual Sound Cards (DVS).  Using the Dante Controller application makes it very easy to setup, configure, and maintain.  After putting all this fun stuff together I decided to visit the Audinate site and check with their online information.  I decided to go through the steps and check out the training.  I did all this just this past weekend.  The end result: I have a Dante Level 2 certification.

I hope to catch up and spend time on some details on why I chose Dante, and how I implemented it.  For now, know that I am a pleased Audinate Dante customer.


Categories: Management, Uncategorized Tags: , ,

NAB Show 2017 Recap

I had to cancel my NAB trip this year!!!!

So there you have it.  The recap of the century.  What was the excuse to cancel?  It was cancel, too.  All plans were made: flight, hotel, appointments, dinner.  All cancelled.  Oh, the excuse:  One of our stations decided to change formats!

First we are on the back half of our facilities move.  More on that someday.  Needless to say, there are many things that are not in place or implemented at this time, so a format change does not make life too easy, especially when one of the staff was on vacation.  Well, we did it.  Format done, but many important meets were missed.

The life of radio sure makes things interesting and last second changes are among them.


PS:  I was involved with putting the first AM all sports station on the air in San Diego.  I now can say I as involved with putting the first FM all sports station on the air in San Diego.

Categories: Equipment

Does Your Transmitter Run on 66 Vac?

October 23, 2016 Leave a comment

Friday and my family was in L.A. I was planning on taking the train up to meet with them as it as Grandma’s birthday, but it was contingent on how the week was heading. South, that is where it was heading!
Sitting at the my desk doing a couple of things the the silence alarm goes off.  Not just one, but two stations collocated.  For that to happen their are only two things that will cause that, main STL to the site or power.  I quickly switch one to the our Aux site, the other I poke at and switch STLs.  Back on.  Off to the site I go.  Mind you I get a low power warning, and both transmitters were on at about 3kW each.  Each transmitter is a Nautel NV20, the very ones that I have posted about before.  In addition a get an high room temperature alarm, so HVAC is down.

I get to the site.  Sure enough the TV station’s generator is running.  I open my door and greeted with hot air.  My lights work, but two of 3 UPSs are beeping away running on battery.  Looking at my surge suppressors, two phases are missing!  Yes, two, and one NV20 was on.  I had turned one off earlier when I switched that station to an Aux site.  Out comes the Fluke and I start measuring the phases.  One is running fine at 120V, the other two are running 66V.  One rack is powered while the other two are on battery.  I make provisions to route power to equipment in a backed up rack preparing for the batteries to die.  I look at the transmitter running.  3.6kW TPO and the status tells me 2 phases are missing.  All I can say is, “wow.”  I had a similar incident years ago with a single phase loss, but this time 2!

Once I had my stuff stable, I noticed the power company, SDG&E, was already outside on the street.  Ah, they were doing something and it went wrong.  Of course the guy in the truck had no update nor did he know where the issue was.  The guys at the fire station, following protocol, had moved their trucks out and were talking with another SDG&E person.  Eventually 3 trucks were up the street.  Within about 45 minutes of being on site, power was restored.

Chalk another one up to Nautel.  The power distribution and fold-back on the NV20s worked flawless.  Needless to say, my HVAC recovered and all other equipment survived the trauma.  Let’s see if the rest of the weekend remains quiet.



Categories: Equipment, Nautel, NV20 Tags: , , ,

Fixin’ Things: Week’s Update

Figured I would try and and get back into the swing of things and update a bit more often.  Mostly boring stuff, but, hey, someone has to do it! This week I have a bit of F.O.R.D. action to mention along with plans to  repair a Nautel NV20.  Also a little fun with NexGen and Sage communications.  A little bit of everything.

F.O.R.D. = Fix Or Repair Daily.  Harris Z10.  This box just sits there and is used on occasion when I do a Nautel update.  As it sits, it fails.  No, it doesn’t need to be turned on, it just fails.  I should clarify, it shows Faults.  Currently 4 when I look at the logs, used to be 5.  Let me remind you, it gets these just sitting idle.  Apparently 2 of the 4, was 3 of 5, are PA modules showing the PS#_OT, or over temperature fault.  Again, it isn’t running.  A little research and some interaction with tech support a couple months ago, I finally ordered the parts through Mouser and did a test repair.  This repair entails the replacement of a capacitor and thermistor.  Easy enough except they are surface mount, so I finally got a chance to experience this type of repair first hand.  All I can say is it takes patience.  Needless to say, the test PA module installed and fault is cleared.  I have 2 more to do.  The other faults I will act on when I get these obvious ones out of the way.  Unfortunately it looks like one of these pending faults corresponds to a dead PA.

But, wait, did I say a Nautel repair is imminent?  Yeah, I guess I pushed the old beast a bit too hard as I prepare for an HD carrier increase.  I began to adjust the HD PA voltage to accommodate the new power increase and two PA modules said they did not like that, and Poof!  I just finished doing some research and I have all the parts to replace them.  I even have the tools!  What did I find out today?  I have version A of the modules and all the latest documents show they are up to C, and they look more modular!  The old ones require de-soldering and then replacement.  At least it doesn’t look to bad.  I also get to perform the modifications that the modules required a few years back to improve efficiency.  All in all, I call this fun.

We us IP to control most everything these days, and this includes how the RCS NexGen communicates with the Sage Endec.  Sure enough, this week I noticed that the RWT did not fire properly.  This is normally caused by the old architecture of NexGen where it just decides to not talk.  After a couple of tests I had to do the old CTRL+Alt+Shift+F4 on the A-server so it would load a fresh database.  Why it decided to start doing this is beyond me.  The other two stations do not have this issue.  I know we will be moving off this system some day, so whatever we use better be IP savvy in running all tests and alerts.

Interspersed among all this fun is Microsoft and Windows 10.  Thanks.  Apparently there is an issue with Win 10 and USB 3.0.  At least with some of our stuff, more specifically Digigram and the UAX-220v2.  Thought he device uses generic Windows drivers, the system will throw a fault and reboot spontaneously.  When trying to track it down, the Event Logs show nothing except the “previous shutdown was unexpected” error.  I finally caught a memory dump and analysed it.  A reference to a USB device was there.  As a test I removed the UAX and that machine has settled down.  I think I will need to experiment with a couple of other USB sound devices and see if it is isolated to Digigram or it is more universal issue.  Sad if it is the UAX as the beauty of it was plug and play with no special drivers.

If you get a long weekend, enjoy!  If not, take advantage of the time you have!



%d bloggers like this: