Fridays. It always happens on Fridays.

In my second week as CIO of a small liberal arts college in flyover country, I had my first test of crisis management, and crisis communication.

It was a Friday afternoon (naturally), and unbeknownst to me, one of our key vendors had scheduled a modification to an existing, mission-critical drive array, expanding its capacity from 2 TB to 4 TB.

Because I am writing this post, you might have guessed that this didn’t go exactly to plan.

It didn’t.

Rather than expanding our existing array two-fold, the entire 2 TB array was wiped out, with a few mouse-clicks.

Poof.

Gone.

In the business, these types of occurrences are known RGEs – Resume Generating Events.

I was hoping that it wasn’t my resume, that was now being queued up in the laser printer.

It was then I then learned, that two of my key team members were out, who normally would be responsible for the care and feeding of these systems. Welcome to Hell, here’s your banjo.

3a12c340c217a874ce620bc2d29eebaa

Next, I sent our campus C-level folks a short, succinct note, indicating the severity of what just happened, with scheduled check in times to keep everyone in the loop. I then set about recalling our key staff, to start the effort of putting Humpty Dumpty back together again.

It took all of us working over the weekend. But, we were able to recover – mostly – everything.

Now, in the history of catastrophes, it’s rarely one thing that causes the world to seemingly crash down around you; but rather, it’s usually a series of escalating, compounding, cascading events that lead up to a final tipping point.

For us, it began with a technician at our vendor’s data center ignoring several important processes, before giving us the literal and metaphorical finger. And, because our 2 TB was now toast, we found out that our most recent backup was over a week old.

An automated daemon process, that was supposed to tell us when our backups had failed, failed to recognize that the backup job itself had been running for well over a hundred hours. The process hadn’t failed in the sense that it had stopped running altogether. But, no one had looked at it to recognize the long running condition, either.

We had just closed our fiscal year two weeks before. Our own personal Armageddon was avoided by this much.

When it rains, it typhoons.

We did live to fight another day. By the hardest.

Wizened, I got to re-learn several important lessons that day; namely, that one should:

  • Never schedule mission critical engineering work, when key support people are out of pocket;
  • Never let automated processes go unmonitored and unexamined; check them – daily.
  • Remember that processes and procedures exist for a reason. If you bet against that notion, then you should be prepared to pay whatever price that entails; and,
  • Neverneverschedule infrastructure work for Fridays.

Because when the world ends, it will be at 4:00 pm on a Friday (probably, Holiday) afternoon.

 

 

 

No One Told You When to Run

And then one day you find,

Ten years have got behind you.

No one told you when to run,

You missed the starting gun.

– Time, Pink Floyd

 

 

We spend a great deal of our formative years, learning the basics of “getting along”:

Sitting quietly.

Being polite.

Following the rules.

All of these are great attributes, to be sure; traits I struggle – daily – to drill into my offspring.

But these traits aren’t always the qualities, that serve us best in leadership.

Who is best served, by sitting quietly, when perspectives and experiences are withheld to solutions being sought?

No one.

Who is best served, when honesty and transparency is required, but politeness prevents a compassionate resolution to a conflict – and instead, prolongs an untenable situation?

No one.

Who is best served, when rules and processes prevent what is right and proper to occur, to correct an injustice?

No one. Or perhaps, only a vanishingly small few.

As a “recovering entrepreneur”, a lesson I learned many years ago, is that if you wait around for someone to invite you to act, you’ll be waiting a very long time.

Because no one is going to tell you when to start living your life. To start contributing.

To start: being awesome.

If you see an injustice, act to correct it.

If you see something that needs doing, don’t wait for someone else to act – do it yourself, or find someone who is qualified, to act.

If processes are impeding what is right to be done, act to change the system.

If you don’t exercise your agency, you will be left at the starting line.

No one told you when to run.

Run.

 

 

 

 

 

 

Audio

Non-Profit Technologists: What Keeps Us Up Nights

Non-profit Technologists

What Keeps You Up Nights?

This is almost a trick question, especially if you work for a non-profit and are responsible for technology in your organization – because the answer is: Everything.

Sure, there are frequently heard answers. Keeping the “trains running on time.” Keeping the “lights on.” Keeping the “main thing, the main thing.”

But – to truly be effective, and innovative – one must look beyond service level agreements, disaster recovery, and crisis planning, and ask – what are the next level challenges that I must overcome as a non-profit technologist?

Technology Refreshes: Non-profits are – and always will be – challenged with funding uncertainty, year over year. As such, planned and dependable technology refreshes – new computers, software updates, new technology purchases – will always be unpredictable and undependable IF your strategy depends only on operating budgets. Innovative non-profit technologists must be versed in grant writing, collaborating with development officers to set up recurring capitalization instruments, such as endowments and chairships. Otherwise, your technology refreshes will always compete – and lose to – higher priority funding, such as compensation and deferred maintenance. Which leads to…

Staff Compensation and Development: When funding times get tough, training and professional development are the first things to go in the operating budget. Recruiting – and keeping – top talent in non-profit organizations is well-nigh impossible. As a leader in the space, you will have to fight over every single dollar; you are not just competing with other non-profits, but anyone funding a tech position. You will have to be able to offer creative forms of compensation, that meaningfully move the needle to affect the quality of life for your staff.

Staying Ahead of the Infrastructure Curve: Large scale investments in our underlying infrastructure (networking, wireless, maintenance, access) demands the lion’s share of our technology planning, and funding. Without adequate infrastructure funding, it’s impossible to stay ahead of the demand curve. Standing pat is no longer an option in this new “Experience Economy”, when always on isn’t an amenity – it’s a necessity; the new normal.

What keeps you up nights, as a non-profit technologist?

Go, and be you.

Audio

The Residential Liberal Arts Elephant in the Room

This Spring, Administrative Officers across the country got to speak to their nervous boards and alumni, to allay fears that their schools wouldn’t be the next Sweet Briar College (which indicated it would be closing at the end of the Summer 2015 term, after precipitously deciding – over a single weekend – to shutter the 114 year old women’s college).

What little is known of the decision to close Sweet Briar is troubling to many liberal arts schools. The barely unspoken fear: Is this us in a few years?

Sweet Briar is an extremely small school: only 513 students. The school has an endowment of $84 million dollars – which isn’t substantially different from many other small residential colleges – but with most of these funds being restricted. Their bond rating was downgraded from “stable” to BBB “negative” by S&P, and reportedly the school hasn’t enough usable cash to pay down debt. And, while their discount rate (the rate representing what students really pay vs. the stated list price) has been rising, there are other small schools with even higher discount rates.

Current trends are against righting the ship. Single-sex colleges are less attractive to students, students are less and less attracted to bucolic liberal arts campuses located in the middle of nowhere, and incoming students are more demanding of amenities and facilities. If you can’t attract students, can’t borrow money to upgrade facilities, and you can’t pay down debt, your options are severely limited.

Is Sweet Briar College the “canary in the coal mine?”

For a certain type of residential college experience, yes. Yes, it is.

In tomorrow’s cast, we’ll discuss possible solutions, and the headwinds that lie ahead for the residential liberal arts college experience, as currently constructed.

Change will come – the question is, will small residential colleges innovate, or abdicate.

Go, and be you.