Fast-time

No, it is not a self help post.

This is a compilation of all the information i dug up, on how you can save oodles of CPU power, if you use the right clock-source for your Operating System.

If you have the time and patience, i recommend you read these excellent articles instead.

  1. 2010 : Overview of linux timer resources
  2. 2015 : Pitfalls of using TSC (assembly instruction directly)
  3. VMWare Time Management (Refers to ESXI 6.5)
  4. 2017 : The slow currentTimeInMillis for Java
  5. 2018: General time management impact from performance perspective (Windows?

Summary – Switch to TSC if running on on modern hardware

That brings up a few questions

  • What do you mean by the right source? Are there many ?
    • Yes there are many. These are the clock sources available to the operating system
      • TSC                (In 2020 this is the best bet on modern hardware).
      • ACPI PMT
      • HPET
      • APIC
      • PIT
      • RTC
  • Shouldn’t the OS decide and pick the right source ?
    • Absolutely it should. The latest & greatest OS’s does so, but if you reached here, you must be either using an old OS, which does not do this automatically, or like in my case, was using outdated kernel parameters or configuration which prevents it from doing so.
  • Why is knowing about clock source important ?
    • You supervise systems and want to know about  (clock drift / performance)
    • You write code & want to know
      • the granularity of your clock ? nano or milli seconds ?
      •  how easy is it for the clock to drift
      • If it is optimal to get the system time very often
    • You deploy virtual machines & want to know how to time behaves in a VM

Practical aspects of the main clock types

ACPI PMT

  • ACPI is a standard used by the system to discover on board devices, much like PCI is used to discover devices attached to the motherboard .
  • PMT ie Power Management Timer is included in almost all ACPI motherboards
  • It is a simple counter that works at a constant frequency
  • It is therefore very reliable.

  • What are its problems?

    • It is extremely inefficient – in fact the pecking order for the time taken to access the clock is ACPI > HPET > TSC.
    • Its resolution is very low and is nowhere near the nanosecond resolution provided by TSC
    • This works on the basis of interrupts and sub second resolution can cause performance issues for the CPU due to frequent interrupts

  •  Advisories against ACPI PMT

    • Redhat advices against using this– On modern hardware ACPI PMT should be used as a clock source of last resort only. See the section on TSC to see TSC related issues which have been resolved & for the hardware types that resolve it.
    • IBM recommends not using this clock setting with VMI Kernels

HPET

  • One of the most interesting aside from TSC
  • HPET is lower resolution compared to ACPI & can protect against timer drift
  • Introduced by Microsoft and Intel around 2005.
  • Uses a dedicated hardware which provides timer interrupts to CPU
  • Is quite complex
  • Older kernel books, used to advice HPET > ACPI > TSC due to CPU frequency shifts causing issues with TSC

TSC

  • The Time Stamp Counter (TSC) is a 64-bit register present on all x86 processors since the Pentium.
  • It counts the number of cycles since reset and is basically a clock counter
  • The CPU instruction RDTSC, RDTSCP (when available) is used to read its value
  • The TSC depends on the CPU clock obviously
  • TSC has traditionally been the source with best efficiency & most resolution but HAD reliability issues described below.

  • Benefits of TSC

  • Nanosecond resolution
  • Least expensive to fetch
  • Least amount of latency to fetch clock data

  • Issues TSC used to have

    • The TSC depends on the CPU clock, which can be underclocked or overclocked as per the current load resulting in time dilation and inaccurate measurements.
    • TSC would run at the frequency of the CPU itself, and if it stalled, TSC would too eg Power management etc
    • SMP – Each core had its own TSC – so the clocks will be out of sync if switch cpu’s – time can even go back

  • Here is how the TSC issues have been solved

    • a constant TSC ticks at the processor’s nominal frequency, regardless of the actual CPU clock frequency due to turbo or power saving states.

      Hence TSC ticks are counting the passage of time, not the number of CPU clock cycles elapsed & is synchronized across cores.

    • an invariant (or nonstop) TSC keeps the TSC running at a fixed rate regardless of changes in CPU frequency.

      “The invariant TSC will run at a constant rate in all ACPI P-, C- and T-states”

    • RDTSCP instruction has been added to ensure that the reordering issue is not present.
    • The Linux kernel makes a host of checks to check if TSC is reliable before deciding to use it

  • Linux CPU flags that indicate the various TSC capabilities above

Flag

Meaning

tsc

The system has a TSC clock.

rdtscp

The RDTSCP instruction is available.

constant_tsc

The TSC is synchronized across all sockets/cores.

nonstop_tsc

The TSC is not affected by power management code.

tsc_reliable

A flag set by VMware to SKIP the kernel TSC reliability checking since virtual environments might create false positives

           Here is how to check for the flags in linux –

cat  /proc/cpuinfo | grep -E “rdtscp|constant_tsc|nonstop_tsc”

How does TSC behave in VMWare

Now that we have established TSC is better than sliced bread, it is quite interesting how and whether the pitfalls called out before is controlled in VMWare, when running under Linux

  • Since ESX 5.x, the hypervisor always maintain the TSC & gets it synced between vCPUs.
  • VMware uses the hybrid algorithm to make sure TSC get synced even if underlaying hardware does not support TSC sync.
  • The trusted TSC flag is set so that kernel does not try and do checks to decide whether TSC is reliable.
  • So in VMWare if this flag is set TSC should be considered reliable
  • The TSC is also handled for VMotion (unlike others??)
  • This is the linux kernel commit, which added the RELIABLE flag

TSC recommendations in the industry 

Note : Here is how you do it, if you are on Linux