[ntpwg] Testing NTP performance

[Greg Dowd]

I think the testing methodology is quite dependent on what you are
trying to observe.  Are you looking for the absolute accuracy of a
client with respect to UTC, accuracy wrt server, relative accuracy
between 2 clients, or a pool of clients, etc.  Or, are you testing a
server's ability to put time on a network?  Are you looking for p-p
error, convergence time, rms.  Are you testing some application which
would add its own error on top?

For my testing, I'm usually testing the egress port of a server for
accuracy with respect to the server clock and jitter.  For that, I use
custom designed ntp "probes".  These probes are pseudo clients,
basically boxes with hardware timestamping synchronized to gps or house
pps and then pointed at a single server and querying at 1 packet per
second.  The probes record the statistics using peerstats and rawstats
which I can then load into a custom statistics package we have called
TimeMonitor Analyzer.  Phase plots, histograms, ADEV, TDEV, minTDEV,
MTIE at your fingertips.  After getting a baseline, I can start loading
up ntp traffic, network traffic, etc and continue to sample.

I am not a big fan of just testing pps signals as they don't tell all
the story.  They give you the end to end performance but don't let you
break down the error into discrete components to analyze.  But, it can
be useful.  It isn't too hard to hack linux these days to capture
timestamps or toggle serial lines.

I have also tested the standard performance of linux clients running
ntpd against our servers.  In that case, one great way is to plug a gps
receiver into the slave, with appropriate refclock, and run the query
against the server notrust.  Log the stats and analyze.

If you can't do that, put a known good ntp server right on the switch
port next to your box and then test the server under test with notrust.

If you are testing the client, the easy way is to turn on stats if
possible.  Not absolutely accurate, but they give good data.  

When testing networks, another thing I like to use are Ethernet taps.
These guys let you "sample" the flow at any point when you are testing
loaded networks (a la g.8261).  Then you can map the packet delay
variation, and its subsequent impact on the client servo.  

I have not been overly impressed with the current generation of network
testers (e.g., spirent or ixia) for their ability to generate packets
with timestamps embedded or to manipulate timestamp fields.  But I think
that is getting better.  You really, really have to watch out for how
the testers are configured when setting up test nets.  It is very easy
to get standing wave patterns in your network that trash the test.
Really easy.  Too easy.  And very time-consuming :-)

  
Greg Dowd
gdowd at symmetricom dot com (antispam format)
Symmetricom, Inc.
www.symmetricom.com
"Everything should be made as simple as possible, but no simpler" Albert
Einstein
 

> -----Original Message-----
> From: ntpwg-bounces+gdowd=symmetricom.com at lists.ntp.org 
> [mailto:ntpwg-bounces+gdowd=symmetricom.com at lists.ntp.org] On 
> Behalf Of STUART VENTERS
> Sent: Friday, May 02, 2008 7:52 AM
> To: ntpwg at lists.ntp.org
> Subject: Re: [ntpwg] Testing NTP performance
> 
> 
> >>Not to get too philosophical, but isn't this a description of the 
> >>precise problem that NTP itself is designed to solve?
> 
> >>Rob Seaman
> >>NOAO
> >>--
> 
> 
> Rob,
> 
> Guilty as charged, there is circular logic here.
>    I'm using NTP's ability to transport time to measure same.
> 
> But perhaps I can get out on a technicality.
>    I'm proposing to use the time transport ability over easy links
>        to measure the time transport ability over a more 
> difficult link.
> 
> I'd be happy to say that the only unit under test is A
>   and so B and C can be 'Golden units'.
> 
> I need to test down to sub-millisecond accuracy.
>   There are servers available for B with 100nS accuracy.
>   There are packet bert testers that could pinch hit as a 
> client C with similar accuracy.
>   These should be sufficient to get us to uS accuracy.
> 
> Again, this seems like a complicated plan, there ought to be 
> an easier way.
> 
> Asked another way, folks here have claimed that NTP runs  
> with particular accuracies much tighter
>    than a human can check by printing out the time on a console.
> 
> How do they know?
> 
> 
> Regards,
> 
> Stuart
> 
> 
> 
> 
> 
> > Greetings,
> >
> > I'm puzzling over how to test embedded NTP implementations.
> >
> > Given a client and server connected through some test network,
> >   how do you accurately measure the difference between the 
> server and 
> > client clocks.
> >
> > If there was a 1PPS test wire coming out of each implementation, it 
> > would be easy.
> >   But unless you can add code to the client, there is usually not a 
> > wire,
> >
> > I need some way to peek into the client and server to see their 
> > internal clocks.
> >
> > I'm thinking the server functionality embedded with the 
> client might 
> > help.
> >
> >
> > Consider the following system with a third NTP box:
> >
> >
> >                        (C)
> >                       CLIENT
> >                    /         \
> >                  /             \
> >                /                 \
> >              /                     \
> >            /                         \
> >          /
> >      SERVER ---------- TEST  ------ CLIENT/SERVER
> >        (B)            NETWORK           (A)
> >
> >
> >
> > We are trying to measure the effects of the test network on 
> the client 
> > in A.
> >
> > Server B provides a stable timing reference for the test.
> >
> > Client/Server A is locked to Server B through the test network.
> >
> > Client C is connected to servers A and B.
> >
> >
> > If the links connecting Client C are simple point to point 
> links, then 
> > hopefully most of the errors in time transport
> >   are due to the test network affecting A.
> >
> > C provides an offset between B&C and also between A&C.
> >
> > It seems like the difference between these offsets  should be the 
> > offset between A&B.
> >
> > This seems like a complicated plan, is there a better way?
> >
> >
> > Regards,
> >
> > Stuart
> >
> >
> > ps: My apologies for an NTP 101 question.
> >     Perhaps someone could point me to where this has already been 
> > sorted out.
> 
> 
> _______________________________________________
> ntpwg mailing list
> ntpwg at lists.ntp.org
> https://lists.ntp.org/mailman/listinfo/ntpwg
>