Follow us on Twitter

On-time PT stats: too blunt a measurement tool?

Humantransit has a thought-provoking blog post on whether measuring “on time performance” is really the best way to gauge the effectiveness of public transport in providing what its users want and need. Here’s a couple of interesting paragraphs:

I have a great deal of sympathy for transit executives trying to deal with on-time performance, because many of the causes of delay are outside a transit agency’s control. Still, there are two major problems with the measures of “on-time performance” that prevail in the industry.

1. They are not customer-centered. They report the percentage of services that were on-time, not the percentage of riders who were. Because crowded services are more likely to be delayed, the percentage of customers who were served on-time is probably lower than the announced on-time performance figure.

2. For high-frequency, high-volume services, actual frequency matters more. Suppose that a transit line is supposed to run every 10 minutes, but every trip on the line is exactly 10 minutes late. A typical on-time performance metric (e.g. the percentage of trips that are 0-5 minutes late) will declare this situation to be total failure, 0% on-time performance. But to the customer, this situation is perfection.

For a while this year ARTA and Veolia placed significant emphasis on advertising their ‘on time’ stats – even though in the early part of the year these statistics were truly horrible for Auckland’s rail services. Yet at the very same time as rail performance was so terrible, we saw rail patronage skyrocket to record levels.

ARTA’s main response to the poor performance stats fell into the traps outlined above – too focused on the simple ‘statistic’ and not focused enough on the experience of the rider. Basically, they made the timetable slower. While a greater percentage of trains now get to their destination “on time” (in the crazy world where on time means no more than five minutes late) because they do this by simply adhering to a slower timetable, chances are that your average Western Line user (in particular) has a slower trip now than they did back when the trains were so unreliable (particularly now that the Western Line’s express trains have been removed from the timetable).

I think the points made in the Human Transit blog post are highly valid when it comes to Auckland – that what we really need to be measuring are statistics that people using the rail system find important. How long are they likely to have to wait for their train? How likely is it that their train will get them where they’re going in the time it should? How long will their trip take? While reliability – which is really all that on-time performance stats measure – is very important, so are other aspects like ensuring consistent spacing between services and trying to get the trains to travel as quickly as possible.

I suppose that the main problem I have with the current focus on “on-time performance” is that it encourages overly slow timetables. This may make the statistics look nice, as it’s pretty easy to keep to a very slow timetable, but in the end the slower you make the train trip – the more likely someone is to choose to drive instead. So perhaps we need to broaden our measurement of how good the rail service provided actually is. Perhaps we need to think more about what really matters to customers, rather than trying to find a simple measurement statistic that perversely makes our trains slower by encouraging an overly forgiving timetable.

On time performance is useful, but not in isolation and not to the cost of everything else (like speed and frequency).

11 comments to On-time PT stats: too blunt a measurement tool?

  • Chris R

    That’s right – it can lead to timetable manipulation, however there has to be some measure of performance otherwise it would be giving Veolia a blank cheque.

    What do you have in mind to replace OTP?

    • I’m not sure exactly what, and I think that OTP has its uses. Perhaps something like “average time to complete journey” might be useful. Then we can see over time whether the trains are getting faster or slower.

      OTP would be OK if we had a timetable that encouraged the trains to go faster, rather than one whose sole purpose seems to be to minimise the chance that a train will be “late”.

      • Matt

        How about “percentage of services running to scheduled journey duration” where the leeway is +/- 10% of the scheduled duration: a service from Otahuhu to Britomart would get about 3 minutes of leeway, a service from Waitakere or Pukekohe would get about 6 minutes. That, to me, captures one of the key pieces of information you’re trying to gauge.
        Another would be “percentage of inter-station journeys slower than scheduled” and, conversely, “percentage of inter-station journeys quicker than scheduled”. The former would help with telling whether the schedule is actually realistic, and the latter with legs that have too much time allocated to them.

  • Ross Clark

    My day job is working as the rail performance analyst for one of the ‘devolved’ governments in the UK, so what you discuss is all too familiar. All the debates you note here, such as the trade-off between faster journey times and a more robust timetable, are ones we have had.

    The easiest thing to do, would be to break out the peak operating stats, which cover well over half your patronage, from the non-peak ones. Here, you would concentrate on am peak arrivals and pm peak departures at Britomart, and from that you would see a much more realistic picture of things. I can guarantee that Veolia would be able to provide that sort of information.

  • Matt L

    I don’t think it is the greatest measurement, of interest I have noted that since the timetable was lengthened slightly the time keeping seems to have gotten worse so that hasn’t even solved the problem.

    Perhaps we need something on the trains to record what is happening so we can look at things like average dwell time, average speed when not at a station etc. That way we can hold the appropriate people responsible as if there is a slow TM who leaves the doors open it can be identified, if there is a slow driver then that can be identified and if the train details are included then we might be able to identify which trains are slower than others.

  • Fullers Ferries has also “slowed” its timetable and advertising to say it’s now 40 minutes to Waiheke, not the always missed 35 minutes.
    It had for a while very patronising little posters on board stating the boat will not wait for you and leave on the dot. When passengers pointed out that they preferred arriving on time more than leaving on time (because ongoing bus/train connections were constantly being missed) the company finally gave in and stated the actual sailing time instead.

  • Chris R

    I don’t want to see drivers penalised because they are “slow”.

    A driver must keep his train under control at all times.
    If driver A has been on the job for years he may be able to “push the envelope” a lot more than than driver “B” who is is just out of OJT and less keen on pushing his train hard; he may even feel that by doing line speed he doesn’t have his train under control.

    • Matt L

      I agree, that’s why I suggested separating out the dwell time from the driving time as that is something that is more in the drivers control. With things like this you would also have allowances for new drivers to get up to speed as it is unrealistic to expect someone to come out of training and be as experienced as someone who has been doing the job for 10 years. In saying that though there needs to be a timeframe for new trainees to get up to speed and if they aren’t meeting the requirements after a set time then their performance needs to be addressed.

      I think that one thing that is needed is a independent audit of things like timetabling so that we know that the times advertised are both realistic and reasonable. I think there is far to much opportunity for those currently involved (ARTA, Veolia and Kiwirail) to manipulate the timetable to make things look better. If we knew the timetable was the best we could expect from our system then it would make the on time performance result more relevant.

  • In Melbourne at the last re-tender of the rail operator they adjusted the definition of what counts as ‘on time’.
    Then the new operator spent six month spruiking their fantastic immediate improvement in punctuality.

    Damned gerrymandering!

  • Chris R

    “I think that one thing that is needed is a independent audit of things like timetabling so that we know that the times advertised are both realistic and reasonable. I think there is far to much opportunity for those currently involved (ARTA, Veolia and Kiwirail) to manipulate the timetable to make things look better. If we knew the timetable was the best we could expect from our system then it would make the on time performance result more relevant.”

    I’ve been saying that since Veolia took over!

    I was told that tiemtabling (as in inter station running times) was nothing to do with the operator and was dictated by ARTA!

    • Matt L

      “I was told that tiemtabling (as in inter station running times) was nothing to do with the operator and was dictated by ARTA!”

      I think it is but it is still in ARTA’s best interest to make the stats as good as possible as after all they report to the ARC

Leave a Reply