Enterprise hardware purchases and the sysadmin effect

The other day during an editorial meeting, the Enable Sysadmin staff discussed the topic of hardware procurement by sysadmins in today's IT landscape. There is so much talk these days around computer hardware, and while much of it is centered around crypto mining or gaming, there are still large conversations around enterprise-level hardware. I wanted to poll our audience to see who had racked a server in the past, much less been involved with hardware procurement. 

My starting assumption was that many sysadmins don't actually have much of a say in the hardware that they operate every day. I tend to see the system administration field so divided up by specialization that most people come in and maintain their designated systems and go home. We have storage admins, network admins, security admins, and so on. What's more, most of the purchasing of hardware is done by "they, them, or the powers that be." Somewhere in the chain, there is an elusive authority with procurement power. I could be wrong. In fact, I hope that I am and that most sysadmins do have a say in their hardware selection and purchasing.

[ You might also like: How to acquire Linux server hardware and put it into production ]

As I think further into the question of "Who does have the purchasing power for new hardware?" I think of the system architects who design the systems that we administer every day. Do they have a say in the hardware used in their designs? I mean specifically, down to the make/model of the equipment. Are they just given a budget and free choice? To answer this question, I reached out to Chris Nicholson.

Chris is the founder of Pathmind and Skymind (two AI startups focused on deep learning and machine perception) and has a long history of being in the room for the kinds of conversations we are talking about today. When I asked Chris about his experiences, he delivered some fantastic insights:

Most enterprise hardware is purchased through a complex process that typically involves several stakeholders. While one person may sign the purchase order, ultimate responsibility and authority can be hard to pin down, because lots of people are consulted in those decisions.

Very often, the IT department head will have final say, and the sysadmin will report to them, or up the chain to them. In a sense, the difficulty of answering the question of who has the real purchasing power is the problem of most tech sales to enterprise. There is usually a cast of dozens or hundreds of people to satisfy, and their needs don't always overlap.

This is one reason why you see "shadow IT" running small workloads on cloud instances, or data science teams running a couple of GPUs under their desks (and keeping their shins warm) -- those teams were somehow unable to get their needs met through standard procurement, especially in a timely way.

In really large tech companies, you'll see functional teams tied to a product or software service bring their needs to the infrastructure teams. The hardware that is procured needs to serve the strategic goal of bringing that product to market or making it better. Those teams might say: "This is the kind of product, service, or functionality we need to provide; here are our ideas about how to do that."

Then the infrastructure teams, which include infrastructure software, will try to figure out which combination of hardware and software can actually meet the requirements of the product team. For example, you might see a software infrastructure team in charge of a Spark cluster, and that Spark cluster needs to run on a certain number of GPUs to ensure that performance specs are met for the functional team. That is, they may have to show they can process a certain volume of data within a certain time limit or give a response within a certain latency threshold. (To add a little more complexity, the software infrastructure teams are usually serving the goals of several functional teams.)

These choices are made within many constraints:

• Performance requirements such as speed, stability, and security
• Budget
• Availability of the hardware (certain GPUs have been famously hard to procure)

Sysadmins have a role in shaping the conversation to ensure that these clusters are stable (they are the ones whose pagers will be ringing at night, and continuity of service for internal and external customers is a keep spec to satisfy).

That affects both the hardware that gets purchased as well as the configuration of that hardware, and the software stack expected to run on it. But sysadmins often do not have final say. They are one of several voices. They often steer the decision within the bounds set by the product teams' goals.

Good IT leaders are taking sysadmins' perspective into account, while also seeking to satisfy other stakeholders in the company.

As I take Chris's experience into account, I begin to see that the question of "who has the purchasing power" isn't as simple as I originally thought. Like most things in a large organization, there are multiple parties involved in major financial decisions. I tell myself that businesses want to purchase as few solutions as possible to meet as many needs as they can. This means that compromises have to be made between teams, and a significant amount of input from each team lead is required to do this well.

Damon Garn, former IT administrator and technical writer at Cogspinner Coaction, offers an additional perspective:

One aspect of a sysadmin's influence over hardware purchases has to do with the size of the organization. In smaller organizations, sysadmins may have more direct control and decision-making ability. The spotlight tends to shine on large companies, but smaller networks make up a huge percentage of the IT world.

I was the administrator for a small organization where I managed the servers, network infrastructure, and end-user platforms. As an administrator, I could buy whatever I wanted within a few simple boundaries: Budget, standardized hardware, and preferred vendor.

• Budget - this one is pretty obvious: Don't spend more than ____.
• Standardized hardware - I have worked with many organizations that try hard to standardize on a single platform for ease of support and management. This is particularly true for operating systems, often placing Linux and macOS at an undeserved disadvantage merely to keep support simple.
• Preferred vendor - Most organizations have a preferred vendor to work with. There may be pricing incentives, and there certainly is convenience. As an admin, I could buy whatever I wanted as long as it came from our preferred server/workstation or network devices vendors.

I made decisions on specific hardware all the time, so long as the purchases fit within the parameters above. There were real advantages. For example, one email address or phone number got me a vendor rep who could deliver hardware quickly, without tedious billing concerns. There are also concerns, such as vendor lock-in or standardizing on less than ideal operating systems.

[ Free course: Red Hat Satellite Technical Overview. ] 

Chances are good that if you haven't been asked for input in a situation like this, it's only a matter of time. As we advance and mature in our careers, our voices and experiences become exponentially more valuable to the leaders that depend on us every day. If you have experience in this area, similar or different from what we discussed today, feel free to reach out to the Enable Sysadmin team.