After-hours support: long nights and a home-made amplifier

Later in this blog I’ll explain the amplifier. For now, let me say that I work full-time in support for Factora. I’m a process engineer, so it’s a good fit. And I’m a focused, dedicated kind of individual, which, you’ll see, ties in with the home-made amplifier.

We’re not all process engineers in support; that’s a strength we have at Factora, that we learn from each other. We share our experiences, formally and informally. There’s quite a range on the support team: other types of engineers, IT specialists, developers, and so on, with a wide range of experience and work backgrounds.

All of us have a deep, deep knowledge of Manufacturing Executions Systems (MES). That’s a necessity.

We’re also experienced in operation, how factories actually work, so that we can use terms that operators understand. It’s a major emphasis at Factora, using terms that the customer understands. The majority of the time we’re talking with someone who knows nothing about IT except that their system isn’t working properly. We use terms that fit in their world, in the operations world. When we talk to people who work in IT, the conversation changes; the terms and concepts we refer to are different.

How is after-hours MES support different than daytime support?

In one way, it’s the same thing – support – but in a couple of critical ways it’s a whole new ballgame:

  • While regular MES support during the day can concern any MES issue, after-hours support is only for an MES-related issue that interferes with production. De facto, it’s stressful. Once the customer calls on after-hours (or is routed to us by their I.T. Center) it’s up to you to troubleshoot and solve the problem. Quickly and efficiently.
  • Working in after-hours, you’re alone, remote, it’s night-time, and as mentioned, production’s on the line. You need to stay calm, think hard, dig deep … yet still know when the smart choice is to call on other Factora experts.

What issues occur?

They vary considerably. Sometimes it’s a piece of cake, solvable with a few quick clicks. Sometimes we encounter a tough issue and spend the entire night on it, working against time. Sometimes we need to contact other groups to assist because MES is not the root cause of the problem; then we help our IT Help Center to forward it to the right group.

What’s an easy issue?

One example of an easy issue is an error performed by an operator. Once the operator asks us to correct it, we can usually fix the issue in the database in a matter of minutes.

What’s a hard issue?

Typically an issue never before seen or recorded, it’s one which requires our highest analytical skills, in combing the log files to ascertain the root cause. In the worst-case scenario, we need to review model design or the mechanics of the actual platform we’ve used (produced by another vendor, such as GE) to generate a better performance.

Here’s how a really hard issue plays out at Factora:

  1. A customer calls in with an issue that is not in our support documentation or addressed in one of our Wikis – an issue we’ve never seen before.
  2. We ask the operator questions and then try to reproduce the error on our side. We then review the error log details to try to ascertain the root cause.
  3. We find it.
  4. We may choose to do a quick patch, in order to get production re-started immediately. That gives us time to consider how best to address the issue permanently.
  5. We ask ourselves: Could it happen again?
  6. If so, what’s the best possible solution?
  7. Can we make a repair to prevent the root cause ever happening again? Or can we make a repair to stop it automatically if it does?
  8. We present options to our client, letting you choose the solution.
  9. We create a Wiki page to log what we did to repair the issue and email it to all relevant team members and customer staff.

Solo in the after-hours

Factora has developed a number of processes to decrease the stress of after-hours support, and thereby optimize uptime/productivity of our customer’s equipment even in off hours:

  1. Support team members are informed if there is any new deployment or new implementation planned at any client during the day.
  2. Whenever a support team member finds an issue which could rise again, with the same client or any other, we make a point of informing all team members, explaining the root cause and the quick solution. At the same time, we continue to work on resolving the issue permanently.
  3. Many Wiki pages have been written by senior Factora staff about how they resolved particularly challenging issues. We regularly refer to these pages when we have similar issues during after-hours support.
  4. We record important issues in a Production Issue list. This list is critical to knowledge transfer.
    • We retain the knowledge of employees who leave the company<
    • Employees returning from vacation consult it upon return to see what they’ve missed
    • Last but far from least, we often begin work on a new issue by searching this list, entering a few relevant keywords to find out what’s already available.
  5. It’s an understood part of being on the team that the on-call person can reach out to other members or senior consultants to ask for help if an issue is particularly challenging.

A focus on production

Nothing matters more than keeping your plant operating. We restore productivity quickly by:

      • The team having a very deep knowledge of MES, individually and collectively. Team members know how MES works in different plants, how other related systems work, and how they connect with MES. Only experienced team members are put on support.
      • We are trained in how to effectively communicate with users/operators, and who to address if MES is not the root cause.
      • Only people who can work alone in the night under considerable pressure are chosen for the support team.

How do we find these people?

We don’t! We train them.

Of course, hiring is a critical component of finding qualified people. As an example, before Factora, I worked in both team and project management with a forestry company. Part of my job was in managing customer support. Before that, I earned my masters in engineering.

Nonetheless, after hiring, or a Factora employee moving into the team from somewhere else, support training is serious and structured. New team members start by studying real, past issues, starting with easy and moving on to harder as they progress. Once they’re ready, they get already-resolved issues to ‘solve’ in a safe environment – a testing center – where their actions won’t affect a customer.

Finally, they graduate to working with customers on actual issues, closely monitored by a more senior member of the team.

Working with the after-hours team

It’s an amazing group, a pitch-in team, even when the issue is covering holidays or family time. And in the case of really tough customer problems, I always feel confident that my team members and senior Factora consultants will help me when I reach out.

Solving a technical problem with a porcelain cup

To conclude, let me tell you a story that I hope will make you smile – the one I referred to in the title, about my home-made porcelain amplifier.

When I first became a member of the (on-call, remote) after-hours team, I worried about my sleep patterns. I’m an exceptionally heavy sleeper, I feared I might miss a customer call. Sometimes there wouldn’t be one for hours, so it was possible to fall into a heavy sleep. What to do?

I’m a technical person, I looked for a technical solution. I searched online, everywhere, for a speaker that would amplify the ring of my phone. But I had no luck at all. I found many that would amplify music, but none that made the ringtone any louder.

So I landed upon the idea of a surrounding echo chamber … aka a porcelain mug (yes, a glass one would have worked, but my phone is big, and I couldn’t find a mug that it would fit in). If I could find a big enough mug to put my phone in, I reasoned, the surrounding walls would amplify the sound and I would wake up.

Now it was time for on-site testing. I began to research mugs. After a number of disappointments, I found one I liked (research had revealed that ¾ of the height of my phone was just about perfect).

And my solution worked. For months, until I learned to rise out of even the heaviest slumber when an after-hours call came through, I made sure my phone was in my porcelain ringtone amplifier before even thinking of going to sleep.