[ntpwg] Testing NTP performance

Sun May 4 21:21:27 UTC 2008

Todd,

Performance criteria are no more appropriate in a formal NTP 
specification than they would be in a TCP formal specification. 
Performance crteria are not in the PTP specification nor in the IEEE 
specifications. What you get from the NTP specification metrics are a 
set of averages and upper bounds. In many cases you can extrapolate 
these values to future expections and estimate future behavior, but that 
is not a matter for specification.

There may be merit in subjecting an implementation to synthetic sources 
that faithfully emulate real phase noise and frequency noise, then 
measure the response statistics. I have done this in several papers and 
the book. The dominant determinant is the Allan intercept. You could say 
at a given Allan intercept the statistics should be such and so. I 
wouldn't bet that every conforming implementation would behave the same way.

You can project nominal behavior by class of application and produce 
cumulative statistics and time series, as I did in the book. These 
statistics typically have long tails, so it might be that the median or 
mean error is mumbleseconds, but that hides the fact that occasional 
measurements may be many times that. I cite a published book called "How 
to Lie With Statistics".

You are free to characterize some applications as I did or concoct your 
own. In any case you must include a disclaimer that the characteristics 
are based on averages with ill defined upper bounds.

Dave

TS Glassey wrote:

> Dave the design wasn't the question - nor was the excellent and 
> innovative work done by you and others in the advancement of 
> asymmetric time management schema development. The question was what 
> are good numbers for establishing minimum performance number's for the 
> composite 'NTP service tool and the computer running it' to meet and 
> how do we characterize the performance of the appliances or computer 
> based NTP service systems available toay.
>
> The intent is to be able to set specific performance and 
> return-performance numbers and to publish them as part of the formal 
> characterization specification.
>
> Todd
> ----- Original Message ----- From: "David L. Mills" <mills at udel.edu>
> To: <ntpwg at lists.ntp.org>
> Sent: Sunday, May 04, 2008 11:04 AM
> Subject: Re: [ntpwg] Testing NTP performance
>
>
>> Danny,
>>
>> With due respect, you guys are missing the point. The NTP design,
>> specification and implementation have very specific performance
>> objectives and metrics. The sources of error have been carefully
>> modelled and justified, by analysis and experiment. You might argue
>> about the specific error budget, but it is carefully exposed both in the
>> literature and specification. Even the vulnerability to Byzantine
>> attacks has been carefully considered and defended.
>>
>> In your question below the measured offset is the maximum liklihood
>> estimate of the client offset relative to the server, while the jitter
>> represents the estimated error of the offset estimate itself. The
>> synchronization distance represents the maximum error due all causes
>> with the assumption that the maximum inherent clock frequency error is
>> bounded. The assumed parameters for phase and frequency error estimate
>> are represented by the limbs of the Allan deviation specific to each
>> installation. There's not much more than can be said abount the
>> probabilistics.
>>
>> There is no quantum physics here, although there might be a dose of
>> long-range depndency on oscillator frequency. You can of course argue
>> that the specified performance metrics are flawed and incomplete. But,
>> the basic metrics called out in the specifidation should in fact provide
>> sufficient evidence for an informed judgement of performance.
>>
>> Time and time again folks have requested absolute bounds on performance,
>> such as required accuracy (sic) of one milliseoond within one minute
>> after startup. I use as example an aircraft autopilot. Is it required to
>> place the aircraft within one meter of the centerling of runway 27-right
>> at Heathrow? Under all conditions? Even if the ILS is not operating?
>> Does the landing gear survive an altitude error of one meter? Is NTP
>> operating correctly if required to slew over one minute and the
>> application reserves two passengers for the same seat or sells a stock
>> before it is bought?
>>
>> In short, there are no definitive answers to your questions, only
>> statistics.
>>
>> Dave
>>
>> Danny Mayer wrote:
>>
>>> Rob Seaman wrote:
>>>
>>>> Not to get too philosophical, but isn't this a description of the
>>>> precise problem that NTP itself is designed to solve?
>>>>
>>>> Rob Seaman
>>>> NOAO
>>>
>>>
>>>
>>> I think that the real questions that needs to be answered are:
>>> 1) How accurate is the estimated offset that the server is using at any
>>> point in time;
>>> 2) What is the error in the calculated frequency that the server has
>>> estimated;
>>> 3) What is the error of the jitter that the server has calculated?
>>>
>>> There are two possibilities here:
>>> 1) The server itself calculates these values so it will have an 
>>> estimate
>>> of the error;
>>> 2) An external estimate of these errors introduces you to quantum
>>> physics issues, the measurement affects the values being measured.
>>>
>>> Does this restate the question better?
>>>
>>> Danny
>>> _______________________________________________
>>> ntpwg mailing list
>>> ntpwg at lists.ntp.org
>>> https://lists.ntp.org/mailman/listinfo/ntpwg
>>
>>
>>
>> _______________________________________________
>> ntpwg mailing list
>> ntpwg at lists.ntp.org
>> https://lists.ntp.org/mailman/listinfo/ntpwg 
>
>