Leadership

Fridays. It always happens on Fridays.

In my second week as CIO of a small liberal arts college in flyover country, I had my first test of crisis management, and crisis communication.

It was a Friday afternoon (naturally), and unbeknownst to me, one of our key vendors had scheduled a modification to an existing, mission-critical drive array, expanding its capacity from 2 TB to 4 TB.

Because I am writing this post, you might have guessed that this didn’t go exactly to plan.

It didn’t.

Rather than expanding our existing array two-fold, the entire 2 TB array was wiped out, with a few mouse-clicks.

Poof.

Gone.

In the business, these types of occurrences are known RGEs – Resume Generating Events.

I was hoping that it wasn’t my resume, that was now being queued up in the laser printer.

It was then I then learned, that two of my key team members were out, who normally would be responsible for the care and feeding of these systems. Welcome to Hell, here’s your banjo.

3a12c340c217a874ce620bc2d29eebaa

Next, I sent our campus C-level folks a short, succinct note, indicating the severity of what just happened, with scheduled check in times to keep everyone in the loop. I then set about recalling our key staff, to start the effort of putting Humpty Dumpty back together again.

It took all of us working over the weekend. But, we were able to recover – mostly – everything.

Now, in the history of catastrophes, it’s rarely one thing that causes the world to seemingly crash down around you; but rather, it’s usually a series of escalating, compounding, cascading events that lead up to a final tipping point.

For us, it began with a technician at our vendor’s data center ignoring several important processes, before giving us the literal and metaphorical finger. And, because our 2 TB was now toast, we found out that our most recent backup was over a week old.

An automated daemon process, that was supposed to tell us when our backups had failed, failed to recognize that the backup job itself had been running for well over a hundred hours. The process hadn’t failed in the sense that it had stopped running altogether. But, no one had looked at it to recognize the long running condition, either.

We had just closed our fiscal year two weeks before. Our own personal Armageddon was avoided by this much.

When it rains, it typhoons.

We did live to fight another day. By the hardest.

Wizened, I got to re-learn several important lessons that day; namely, that one should:

  • Never schedule mission critical engineering work, when key support people are out of pocket;
  • Never let automated processes go unmonitored and unexamined; check them – daily.
  • Remember that processes and procedures exist for a reason. If you bet against that notion, then you should be prepared to pay whatever price that entails; and,
  • Never – never – schedule infrastructure work for Fridays.

Because when the world ends, it will be at 4:00 pm on a Friday (probably, Holiday) afternoon.

 

 

 

Advertisement

No One Told You When to Run

And then one day you find,

Ten years have got behind you.

No one told you when to run,

You missed the starting gun.

– Time, Pink Floyd

 

 

We spend a great deal of our formative years, learning the basics of “getting along”:

Sitting quietly.

Being polite.

Following the rules.

All of these are great attributes, to be sure; traits I struggle – daily – to drill into my offspring.

But these traits aren’t always the qualities, that serve us best in leadership.

Who is best served, by sitting quietly, when perspectives and experiences are withheld to solutions being sought?

No one.

Who is best served, when honesty and transparency is required, but politeness prevents a compassionate resolution to a conflict – and instead, prolongs an untenable situation?

No one.

Who is best served, when rules and processes prevent what is right and proper to occur, to correct an injustice?

No one. Or perhaps, only a vanishingly small few.

As a “recovering entrepreneur”, a lesson I learned many years ago, is that if you wait around for someone to invite you to act, you’ll be waiting a very long time.

Because no one is going to tell you when to start living your life. To start contributing.

To start: being awesome.

If you see an injustice, act to correct it.

If you see something that needs doing, don’t wait for someone else to act – do it yourself, or find someone who is qualified, to act.

If processes are impeding what is right to be done, act to change the system.

If you don’t exercise your agency, you will be left at the starting line.

No one told you when to run.

Run.

 

 

 

 

 

 

Unforced Errors

Nothing in professional life is more frustrating, than dealing with problems at work.

That is, of course, unless you are dealing with problems of your own making, that weren’t problems to begin with.

Without “going topical” – or political – it’s safe to say, that unforced errors are front and center in conversations and gatherings around water coolers across the country these days.

What causes us, as leaders and as professionals, to commit damaging – and sometimes fatal – missteps, when these kinds of errors are almost 100% avoidable, by their very definition?

  • Overconfidence.
  • Hubris.
  • Haste.
  • Expediency.
  • Tone-deafness.
  • Failing to recognize our fallibility and – often – our mortality.

Over the years, I have seen the potential for, and realization of, unforced errors almost every working day in my life as an administrator.

Projects scheduled for execution, during times when there is no margin of error should things go south. Skunk work projects created with no backup or documentation, but deployed in mission critical initiatives. Decisions made without the knowledge or consent of stakeholders involved or affected.

In almost every case, these issues could have been avoided through the intentional practice of reflection, consideration, consultation, and communication.

Reflection – is what I am about to do, in the best interest of those involved?

Consideration – have I anticipated the consequences and fallout over what I am about to do or say?

Consultation – have I discussed the action I am about to undertake, with the constituencies who have an interest in its outcomes? And, have I taken into account their perspectives and opinions, in the formulation of my action plan?

Communication – have I transparently and properly communicated the purpose and intent of the action I am undertaking, so that the benefits, risks, and rewards are clearly understood by all involved and affected, and have I created a space in which communication of unconsidered affects, or dissenting viewpoints, may be heard and accommodated?

Even with careful attention to the practices listed above, one can still find themselves embroiled in self-made dumpster fires.

We’re human.

Sometimes, it is our unguarded moments and throwaway comments, that are the rocks upon which careers and lives are dashed.

Even so – if we are present, intentional, and disciplined in our approach and daily practice as professionals, we can – at the very least – be responsible, accountable actors in our decision making, and its outcomes.

As my grandfather used to say – “no need to borrow trouble – there’s enough to go around as is.”