There is a wealth of information about supplier relationship management on the Internet, but I can offer up some KPI’s I’ve used in the past to help evaluate IT provider ongoing performance:
(1) Projects delivered on time – does the vendor commit to delivery dates during project road map negotiation, and adhere to the dates? If there is a regular progression of slipped deadlines or missed deliverables, then the vendor likely won’t get the best score in this area.
(2) Staff training was effective – this is something commonly overlooked in negotiations, but critically important for knowledge transfer to staff if the supplier is expected to transition ownership to the company at some point in time. Did they prepare effectively? Were communications for signing up for the training laid out clearly? Was instruction provided clearly and competently? Did the vendor provide surveys for the trainees to fill out, and if so were the survey results shared and acted on to improve? Was the staff effective and competent on the products once they took ownership of them? All good questions, and it all plays into the overall score.
(3) Products met usability standards – another common mistake in the overall contract negotiations is leaving out the company definition of usability standards. Many times the standards will be something open to wild interpretation, like “make it easier to do business with us”. Keep in mind that one person’s “easier” can be another person’s “more difficult”. It’s really important to get out on the street and get to know your customers to determine what their expectations are relating to usability improvement. Is the vendor doing this? If they don’t know your target audience, then it could be that the vendor is trying to grand-stand and create something that they want to go down in the history books for. It may not make a difference at all to the bottom line of your company. Become a member of the Usability Professional’s Association to get a wealth of information related to this discipline.
(4) Application, database and/or infrastructure architecture delivered is scalable, maintainable, and extensible – The question to ask here is whether or not the supplier delivered a scalable, maintainable and extensible system to the contract’s specifications. If the contract, or subsequent artifact to which the contract refers, did not include specifications or standards to which the system would be built, then the entire topic is open to interpretation, and the waters are muddied forever. If the supplier didn’t include a section in the contract that talks to these topics, then I would simply put them down at the worst score. Suppliers should know enough about what you are trying to accomplish by at least asking the right questions when the project is being defined, particularly if they want to keep doing business with you.
(5) Inter-application integration met or exceeded required standards – Here is an aspect that can’t be overlooked in most IT shops of any size and age. If you have dozens, hundreds or even thousands of applications operating your business, then you must select suppliers that know how to operate in complex computing environments. I’ve never seen inter-application requirements in a contract, but I have seen them (and developed them myself) in specification documents which are referenced in the contract. Without this documentation, and a deep partnership between the supplier and your IT staff, it will be difficult for either party to be successful. So, in my experience the themost important things to measure here are supplier competency in complex computing environments, a well-written specification with expectations, and how well the supplier partners with IT.
(6) Provider staff competence – don’t underestimate the power of competent specialists. I’ve not met anyone in the field of IT yet that can do it all. Specialists are the people that always tend to get to the root of all problems, patch them, and follow up by putting long-term solutions in place.If your supplier is a small firm with a big contract, make sure they know how to obtain specialists should the need arise to leverage one. You don’t want them to run into a brick wall for several weeks while they are running up your bill. I’ve experienced this before, and it’s not a pretty site. Also, make sure your staff is competent at identifying and alerting you to gaps in the supplier’s knowledge. If your team can help identify the gaps early, then you can make sure your project managers are adequately mitigating the risk this brings to the project.
(7) Provider understanding of our product roadmap (vision capability) – Trusted suppliers who have been deemed business partners will need visibility into your product or platform road map in order to operate at their peak efficiency. Once they gain visibility into the road map, its up to you to listen to their comments about it. If they just go with the flow, then they are not being strategic. My expectation is that a supplier will want to understand your road map, and provide you with a variety of options, thoughts, and competitor/industry comparisons to show how they can add value by helping to formulate the vision. When I’ve engaged suppliers that have a deep understanding of how to do this, they really shine and add real value to help the organization grow.
(8) Industry reputation on social networks – what better way to determine how a vendor will interact with your company than to hear it from their other customers? Not just their hand-picked reference accounts, but the general public. If the supplier constantly gets good commentary on social networks about the quality of their work, then its likely going to be a safe bet to at least try them on for size. And, ask questions on the social networks that are pertinent to the type of engagement you would have them do, particularly when initiating a new engagement with an unknown supplier.
(9) Overall capability – This can encompass a number of subject areas, but in general I want to hire a supplier that can meet all of the immediate needs of the project equitably and under the deadlines set by the company. If the company can meet all of the project needs, I would typically give it a score of 5 (out of 10) and if they can go way beyond all expectations of what we may have a need for in the future, the score would slide up to 10. It’s important to note that overall capability can include things like strategic planning, writing specifications, application development, QA, database modeling, infrastructure design, hosting, and a whole lot more.
This is not an exhaustive list, but it can help you create a group of solid starting-point metrics that can be measured on a vendor scorecard if you need to create one. You can apply weighting to them if you want to add multipliers, but I seldom have needed to do this. It’s a good idea to collect both qualitative and quantitative data about your projects, so that in the end, a decision of whether to keep the supplier for future projects is supported by data. Just make sure the data you collect can result in an actionable outcome, such as vendor negotiations, consideration for further work, or release from duty.
Success Strategies 