Roughly Accurate or Precisely Wrong – How to deal with S&OP Forecast Error

This journal article follows up on my April 2016 article If You Can’t Measure It, You Can’t Manage it! In that article I reviewed the mechanics of setting tolerances and measuring performance to plan for the Booking Plan, Shipment Plan and Supply Plan. Using these measurements as a starting point, I will go through the importance of accuracy rather than precision and then take a quick look at the impact of variations that are within tolerance

I have written this article as a story drawing on two of the principle characters from my book “Sales and Operations Planning – How to Run an S&OP Process that Everyone Understands”. Jim, the CEO of ToyAuto, is currently implementing S&OP. Doug the consultant is coaching him on his journey. This story takes place in Jim’s office. Jim’s son Billy makes a cameo appearance. Hope you enjoy it as much as I enjoyed spending time with these characters again.

Duncan

For a printable version of this document: click here

Scene: Jim’s Office. Jim and Doug are sitting at a table and reviewing his monthly S&OP plan.

Jim – Doug you said we need to get the plan roughly accurate rather than precisely wrong. I’m not sure I get the difference between accurate and precise.

Doug – Your son Billy still plays hockey doesn’t he?

Jim – Not just play, he eats, drinks and sleeps hockey.

Doug – Well let’s use a hockey analogy then. The net is 4 feet high and 6 feet wide, measured on the inside of the pipes by the way. The posts are 2 and 3/8 inches in diameter.

Jim – OK, if you say so.

Doug – Let’s say Billy is practicing his shot. If he continuously rings the puck of the left post we would say he is precise, he always puts the shot in the same place, even though he doesn’t score. On the other hand if he puts the puck anywhere inside for the 4 x 6 foot area that is the net we would say he is accurate, he scores.

Jim – That makes sense. The net is a way bigger than the post but you don’t score hitting the post over and over again. You gotta put it in the net.

Doug – Right. As long as you hit the net you score. The size of the net represents the tolerance from center point. Of course the goalie adds a bit of a challenge but let’s keep it simple.

Jim – OK so how does this apply to forecast accuracy?

Doug – If we set an upper and lower tolerance for the forecast, and if the actual results are within this tolerance, then the forecast is accurate, we score. Remember we are talking about the family forecast in monthly buckets, not the weekly SKU forecast. This reduces a lot of the noise in the plan.

Jim – Yeah, Tom Wallace and Bob Stahl nailed that one in their S&OP How-to-Handbook. It is amazing how much time we used to spend in their Suicide Quadrant. So how do we set the tolerances?

Doug – My method is not very sophisticated but experience has shown it works. I take the comparison of the forecast one month prior to the actual results for the last 12 months. I then eliminate the highest point of error (1 of 12 for the 12 months in the year) and set the tolerance equal to the next biggest error.

Jim – Huh?

Doug – (opening up his laptop) Let’s look at an example. The following chart actually has 2 years of data, which is a luxury we don’t usually get. Notice that in the first year there is one point with an error of -24% and then in the second year there is a point with an error of 20%. If we drop these two points all of the rest of the data falls within plus or minus 17%. I like round numbers so let’s call this plus or minus 20%.

Jim – OK, but it looks pretty crude.

Doug – It is but there really isn’t a better way to do it. You are lucky if you have 12 data points when you start out, hardly statistically significant.

Jim – OK I buy that, but you know some of components have three month’s lead-time. We need to know the error based on the forecast three months ago, not one month.

Doug – Yes, if you have a long lead time, you need to calculate your tolerance for month 2 based on the forecast 2 months prior and for month 3 based on the forecast 3 months prior. You would then need to map this against the cumulative lead-time profile for the family and calculate your future flexibility per month. For now let’s keep it simple and deal with the one month error, once we get that working we can add some complexity.

Jim – For sure. So what’s next?

Doug – (opening another worksheet) Look at this family graph for bookings. The grey area is history and the white area to the right is the plan going forward. In the history section the actual results are compared to the upper and lower tolerances based on the plan set one month prior. In the future section the upper and lower control limits are based on the last plan.

Jim – So we have been in tolerance so far although it looks like we were right on the limit in December. Going forward the changes to the plan are in tolerance except for September.

Doug – Right and we have a lot of time to prepare for that.

Jim – So if we are between the red lines we are accurate. Not necessarily precise or on the center line, but accurate.

Doug – You got it.

Jim – But plus or minus 20% on bookings seems like a lot.

Doug – It is what it is. Maybe there are things that can be done to improve the accuracy but today you need to run your process based on demonstrated capability. That’s where the buffers come in.

Jim – Do you mean the buffers we have talked about, backlog, inventory and upside capacity flexibility?

Doug – Exactly. Let me walk you through a real life example. I will simplify some of the variables to make the point clearer. Company X has a build to order family. The market expected lead-time, and in this case their open backlog, is 4 weeks. This means any new order would be promised for week 5. Their process time through the plant is 2 weeks. The lead-time for components is 6 weeks resulting in a total cumulative lead-time of 8 weeks. They have a policy of allowing two Saturdays of overtime per month and keep an extra 2 days of component inventory to support this overtime should they need it.

Jim – OK, let me write this down.

Doug – Now let’s say the average flow rate per S&OP period (4 weeks) is 100 units. Ideally, for this family, they would book 100 units per month; build 100 units per month and ship 100 units per month. However, we have a plus or minus 20% tolerance on bookings. What would happen if they booked at the high side of the tolerance – 120 units?

Jim – OK, I've got that as well. Now, let me see... Their month one production and shipping plan won’t change as they will be building the backlog that they have already committed to. In month 2, where the month 1 bookings are being promised to ship, they could increase production by 2 days. Let’s see. If there are 20 days in a month, two extra days would be 10%. They could get an additional 10 units out in month 2. They would still be 10 units short though.

Doug – And where would those 10 units go?

Jim – Right! They would be added to the backlog. The backlog would get an extra 10 units or 2 days. They would go from a four week lead-time to a four week and 2 day lead-time. That’s such a small bump nobody would even see it happen. Hmmm….there is no big issue going to the top of the tolerance if you have the buffers set up to deal with it.

Doug – Right. Now what if the opposite happened and they booked on the low side of the limit.

Jim – Let’s see. The plan is 100 but they book 20% under, that would be 80 units. They could take Friday’s off for a month, but that is an expensive alternative….or, or they could pull that backlog ahead. They have 4 weeks of backlog, they are producing about 25 per week (100/4), about 5 per day, so they would need to pull 4 days of backlog ahead to maintain the production plan. If the customers won’t take these units then they would go into finished goods inventory for about a week. Probably not a big deal, you wouldn’t notice the impact of the 20% under booking.

Doug – There you go. The buffers are based on the tolerance and if you set them right and stay within tolerance things run pretty smooth. Of course we have simplified this example but if you work through the details the logic works. It is just math.

Jim – Yeah I can see some issues with bias if the error goes the same a few months in a row , that probably increases the buffers, and we talked about the accuracy two, three and four months out for families with a longer cumulative lead-time. There are details to work out for each family but I get it. Hmmm…say what happens in that one month per year when we are out of tolerance?

Doug – Yeah things don’t always run smooth. That is when you have to pull out the stops. It could mean asking for extra overtime, expediting components, even airfreight and potentially extending lead-times beyond where you would like them to be. The key is that this should be the exception, not the rule.

Jim – Yeah for us that reaction is our norm not the exception. Let me recap the steps and make sure I didn’t miss anything:

Doug – You’ve got the basics.

Jim – It seems too simple

Doug – It is just math. In reality it may be a bit more complicated than the example we used but the math doesn’t change.

Jim – Well let’s get on with it!

How effective is your S&OP?

Evaluate your process and get recommendations.

At DBM Systems, our consultants have over 20 years of experience providing S&OP leadership to businesses worldwide. We equip teams with coaching and the tools needed to run an effective S&OP process. Learn about our process and unlock the power of S&OP in your organization.