Intelligent Business

Measuring performance: Big Brother is watching you!

Currently, I am working on a big implementation of MS Project Server for operational and tactical planning. With this tooling, the organization that I am working for wants to create insight into many aspects of one of their programs. Hours spent, resource allocation, forecasts, project statuses, resource calendars: they all are being mentioned as part of the scope of our project. With the recording of the data of many projects and resources, they hope to gain control over the progress and quality of their entire program. An interesting endeavor!

In this phase of the project, we are collecting data from the first projects and sub-programs. And indeed: over allocation of resources and tasks that overlap become apparent! Tomorrow, I will start creating reports based on this data, with the functionality of Microsoft Analysis Services. But I am wondering, what will these reports show and cause? What will the organizational impact and acceptance be? It seems to me that these aspects are currently being forgotten…

In my opinion, introducing a tool for planning is not often seen as a form of performance measurement, while it actually is! You are writing down how many hours you expect to work on a certain assignment and after a while you are judged upon the hours spent and the Estimated Time to Complete. Seeing the implementation of, for example, MS Project Server as a form of performance measurement leaves me with two major question marks.

1. How do we make sure that we measure the correct things?
One of the big problems in performance measurement is that it creates wrong incentives for people to act upon. If one department is for example doing everything in their power too increase their profit, including the charge of higher internal rates too other departments, they are reaching their target at the cost of others (Hood, 2003). One example from literature comes from Schachter (2008):
“Indian and Northern Affairs Canada, the federal department that handles first nations issues, told The Globe and Mail last week that the government had won a victory by settling 54 land claims in the fiscal year that ended yesterday. This beat the 12-month target of 50 set by former Indian Affairs minister Jim Prentice, and was miles ahead of the ministry’s long-term average of settling 14 land claims a year.” (Schachter, 2008, p.1).

Of course, many methods exist to balance the measurement of the performance of your organization in order to prevent these problems. An example is a statement that I found online: “It should be apparent (…) that performance should be measured in a manner that is appropriate for the specific organisation.” (Accaglobal website).  But in the project that I am currently working on, we are not using these methods to determine which variables we use in our tool. Do we really want to steer upon the ratio hours planned / hours spent, or do we need to balance this with steering upon quality of deliverables? If any of you have suggestions as to how to implement performance measurement techniques in a planning tool, please let me know!

 
2. How do we handle the resistance within the organization to the new way of working?
Considering that performance measurement is known for causing a lot of resistance in organization, the same can be expected for the implementation of a planning tool.:
“As with any other organisational change management program, implementing a performance measurement system will encounter resistance especially in large bureaucratic organisations.” (Wikipedia).
The way of working and the way the steering happens will be different for all involved. Not everybody likes to have someone looking over their shoulder, especially when these measurements come back to ‘haunt’  you. Many different methods apply when change management is needed, but are they always considered when the implementation of a planning tool is started? In the project that I am working on, this component is left aside.

It is time to start using the many methods and approaches that are already thought of for performance measurement for the implementation of planning tools, especially because with these tools you actually measure performance!

Literature

  

Add comment December 16th, 2008

The BI Business Case

As I discussed in my last article, Intelligence is not only about data and information, but also incorporates the intangible tacit knowledge from your employees. As you can see in the presentation that was uploaded recently, there is a big theoretical reasoning for why an implementation of a Business Intelligence solution should also include actions to promote the sharing and use of tacit knowledge. But how can we quantify the (potential) success of Business Intelligence, and how do we show that a BI implementation that also includes tacit knowledge is better for an organization? In other words, what is the BI Business Case?

The impossibilities for a BI Business Case

Especially in times of economical recession, each new project in an organization has to show a clear Business Case. How many benefits is this project going to create? A Business Case is, as defined by the Prince 2 Project Management method: “The information about costs and benefits of the project and with this the justification of the start-up and continuation of a Prince 2 project” (van den Akker, 2002). This Business Case is leading during a project, and will be used for evaluation of the progress and the results. In order to describe the costs and benefits data and information is needed, including forecasts.

Of course, an implementation of Business Intelligence should also be started with clear goals and potential benefits. Business Intelligence is meant to support decision making in an organization, by gathering data and information from within and outside of the organization and organizing this into management reports. But how do we measure if the decision making has indeed improved? How do we quantify decision making? We could never see the difference between the result of the decision that was actually taken and the result of the other option that was there. So how do we know which decision was the better one? How do we forecast future benefits from this ‘better decision’?

Also, when tacit knowledge is added as a point of attention to the implementation of Business Intelligence, this brings an extra factor to the table. Tacit knowledge in itself is a form of knowledge that cannot be made explicit and therefore not be measured. The use of tacit knowledge in decision making can thus not be determined, it is even so that tacit knowledge probably plays a role in every decision that we make but that the amount of influence can differ per decision. Measuring an improvement in decision making based on both data and information, delivered by BI tools, and tacit knowledge seems like an impossible goal. This leaves a possible implementation of Business Intelligence with a big problem: a quantitative Business Case can hardly be created.

The possibilities for a BI Business Case?

Brockmann & Anthony (2002) seemed to have found a solution. They investigated the influence of tacit knowledge in decision making on the performance of organizations. What better Business Case could we wish for? The authors suggest that decisions on a strategic level are often “ill-defined, unstructured and lacking in precedence”. (p. 437).

Tacit knowledge is just that, and can therefore contribute to this type of decisions. Since there is little or no attention for the use of tacit knowledge in organizations, this can be an extra advantage in strategic decisions.

The authors base their assumptions on the positive relationship of tacit knowledge to decision making on research by Wagner (1987), Wagner and Sternberg (1990). These authors investigated that the use of tacit knowledge has a positive influence on managing tasks, managing oneself and managing others. Also, there was investigation on intuition (compatible with tacit knowledge) from Parikh et al. (1994), Agor (1986a) where it was stated that “faster and higher quality decisions are made when managers rely on their intuition” (Brockmann & Anthony, 2002).

This all sounds like a perfect Business Case. But if we look into the methods that were used to determine whether a person had used his tacit knowledge, we see that we are talking about questionnaires or cases that are presented to participants. Considering my previous arguments on the impossibilities of measuring the amount of use of tacit knowledge and measuring ‘improved decision making’, these methods are not giving us any insight and certainly no reliable figures on future benefits of a BI implementation.

A different kind of Business Case

A quantitative Business Case, as described in the Prince 2 Methodology (van den Akker, 2002), is not something that we can apply onto an implementation of Business Intelligence. This implementation is complex and has many facets, as discussed in earlier articles, and can therefore not be captured by numbers about costs and benefits. So how do we show that an implementation of Business Intelligence actually has value? Well, a Business Case does not always have to contain figures that will show an increase of business results to be valid. There are examples of a Business Case based on qualitative statements and not so much quantitative ones.

During a BI implementation one could for example organize a quality audit of the decision making in an organization before the start of the implementation. At the end of the track, a similar audit has to be performed to see if the decision making has improved in the view of managers and their employees. These are very subjective criteria, but these are the only criteria that you can actually use to measure if your BI implementation has been successful.

Measuring the amount of use of tacit knowledge in decision making is, in my opinion, methodologically impossible. But a focus on the use and sharing of tacit knowledge (for example by looking over the shoulder of a colleague) could contribute to the feeling of improved decision making. So what is the answer to the question that was posed in the beginning of this article: what is the BI Business Case? There is definitely a Business Case for BI, but in the preparation of an implementation it should be proven in a different way than a traditional Business Case. With a non-traditional implementation of Business Intelligence, with extra attention to tacit knowledge, there is also a need for non-traditional Business Case!

Literature

• Agor, W. H. (1986a). The Logic of Intuition: How Top Executives Make Important Decisions. Organizational Dynamics, 14(3), 5 - 29.
• Akker, Ir. A.G. van den (2002). Prince 2 compact. Methode voor Projectmanagement. Lagant Management Consultants BV.
• Brockmann, E. N., & Anthony, W. P. (2002). Tacit Knowledge and Strategic Decision Making. Group & Organization Management, 27(4), 436 - 455.
• Parikh, J.; Neubauer, F.F. & Lank, A.G. (1994). Intuition: the new frontier of management. Cambridge, MA: Blackwell.
• Wagner, A.K. (1987). Tacit knowledge in everyday intelligent behavior. Journal of Personality and Social Psychology, 52, 1236 – 1247.
• Wagner, A.K. & Sternberg, R.J. (1990). Street Smarts. In K.E. Clark & M.B. Clark (Eds.). Measures of leadership (pp. 493 – 504). West Orange, NJ: Leadership Library of America, Inc.

  

Add comment November 5th, 2008

BI projects are just a start…

How many times have you attended a meeting where the other attendants brought their own Excel-sheet showing conflicting figures? Probably way too many times… You wish for everybody to use one source, one definition, and maybe even one report or dashboard for their information needs in this meeting. Eventually a small business intelligence project is started to make some of these reports, but along the way you discover that the information quality is not okay, sources have to be integrated, uniform definitions have to be made and so on. The project scope is creeping and the budget is overrun… Familiar?

Especially when the BI project is initiated top down and organized centrally, information between different information islands (applications, business units or processes) have to and will be shared and exchanged. Both socially and technologically this can enable many consequences in the organization that are all too often not anticipated in the beginning of the project. There is more to the eye than meets the eye…

This is where information logistics come in. A couple of years ago I wrote a master thesis on the role of information in supply chain management. As you might know information is exchanged between buyers and suppliers in the supply chain in order to move goods as efficient and effective as possible to the end-consumer [1].  Besides efficiency and effectivity, exchanging information in a supply chain has many more consequences, but also needs some prerequisites… To start with the exchange of information between buyers and suppliers is only done when a certain amount of trust and collaboration is in place between the involved parties.

While on another level, it is here I see an analogy to aforementioned information islands in an organization that kick started the BI project. Just like organizations in a supply chain, information islands in the organization can be considered as business entities that exchange information in order to operate as efficient and effective as possible. The model I developed for my thesis can be put to good use here.

Namely, these islands will only share information with each other when they have a collaborational relationship and trust each other [1,3]. When they exchange information, transparency is created between them (and possibly other business units). Another side effect is that so-called white spots appear. Some information is not available, or of poor quality. To improve on this, information has to be integrated in uniform definitions or by introducing organizational or industry standards (such as XBRL) [1]. In addition, this can result in information systems that are integrated (in this case for example a datawarehouse, datamart or reporting layer) [2]. Because organizational units are gearing their activities to one another as a result business business process integration takes place which in turn will lead to increased collaboration and trust (in each other and the information) [3]. In doing so a reinforcing loop comes into existence.

So, going back to the BI project, why is it more often than not more encompassing then anticipated? Simply, because it is often not anticipated that initiating a similar project not only creates centralized reports, but as information is the lifeblood of an organization it has both technological, social and organizational effects.

Please let me know if you would like more information on the model or the thesis.

[1] Lee, H. L. and S. Whang (2000). “Information Sharing in a Supply Chain.” International Journal of Technology Management 20(3/4): 373-387.

[2] Aubert, B. A., B. Vandenbosch, et al. (2003). Towards the measurement of process integration. Scientific series. Montreal, CIRANO (Centre Interuniversitaire de Recherche en Analyse des Organisations.

[3] Premkumar, P. K. (2000). “Interorganization systems and Supply Chain management: an information processing perspective.” Information Systems Management 17(3): 56-69.

  

Add comment October 29th, 2008