A Journey into Time – Thin Light

This post is about how misconfigured time can affect the Linux operating system and lead to unexpected results. Also I’ll share some thoughts on how to get it right. In this specific example, the Linux distribution is CentOS 7.

Components Involved

RTC – real time clock. It is sometimes called hardware clock, can be set in BIOS setup screen or from the OS.
System clock – the software clock maintained by OS kernel.
chrony – successor to ntpd that keeps the system clock in sync.
memcached – software that keeps its own time.

What Went Wrong

We rebooted several memcached servers to replace their motherboards. When they were back online, memcached clients could successfully set keys in the nodes, but were not able to get anything out of them.

How it Went Wrong

stats of those memcached nodes showed that cmd_set and get_misses steadily increased, but get_hits and bytes always kept the same.

That made me look into the usage pattern of the memcached cluster. Luckily that was a new cluster and the only usage of it was to put in keys that expire in 10 minutes.

Intricacies of Memcached

Memcached is a little special in that it only reads time from system clock once on startup, and maintains its own clock thereafter.

And memcached’s protocol allows you to set the expiration time in seconds (up to 30 days) or as a Unix timestamp – a single field that could have multiple meanings. That seems to be an anti-pattern to today’s design principles. However, this multi-purpose field is directly supported by spymemcached’s implementation.

Then I took a quick peek at the clock that memcached secretly holds:

date -d @`echo stats | nc {memcached_address} {memcached_port} | grep "STAT time " | cut -d" " -f 3`

It showed that it’s 1 hour faster than system lock. We used a wrapper around spymemcached and to eliminate the ambiguity of expiration time, we made set() method accept only java.util.Date. So when the client set a key that would expire in 10 minutes, the server thought it had expired 50 minutes ago and evicted it immediately.

Messy Time

The error in memcached’s clock could be caused by error in system clock, before chronyd made it synchronized with time servers. How did system clock went wrong? Here’s the relation between hardware clock (or real time clock, RTC) and software clock:

The Linux kernel keeps track of time independently from the hardware clock. During the boot, Linux sets its own clock to the same time as the hardware clock. After this, both clocks run independently. Linux maintains its own clock because looking at the hardware is slow and complicated.
https://www.tldp.org/LDP/sag/html/hw-sw-clocks.html

So the hardware clock must have gone wrong. Running timedatectl revealed that. Wait – I had rtcsync directive in /etc/chrony.conf. The hardware clock should have been corrected, shouldn’t it? timedatectl warned at the end:

Warning: The system is configured to read the RTC time in the local time zone.
      This mode can not be fully supported. It will create various problems
      with time zone changes and daylight saving time adjustments. The RTC
      time is never updated, it relies on external facilities to maintain it.
      If at all possible, use RTC in UTC by calling
      'timedatectl set-local-rtc 0'.

Now I must confess that the servers were configured not in UTC, but in a local time zone. To make it worse, the local time zone is a DST time zone.

As timedatectl warned, we configured the system to read RTC as local time. RTC itself is like an old-style wall clock and is not timezone-aware. The kernel does calculations as needed when it reads or writes the RTC.

The Fix

So the fix was as simple as running what it had suggested:

timedatectl set-local-rtc 0

In my experience a reboot is needed to make it effective. Note that on this reboot, memcached can still get a wrong time from system clock. Rebooting memcached after time is synchronized can be a fix.

More Notes

rtconutc in chrony.conf

As far as I know (briefly looked into chrony’s source) This directive only have effect when rtcautotrim is enabled, or when trimrtc is run from chronyc.

How rtcsync works

rtcsync directive only lets chronyd set a status flag when invoking system call to adjust system clock. The effect can be seen in the output of timedatectl as:

NTP synchronized: yes

Or in the output of ntptime as:

ntp_adjtime() returns code 0 (OK)
...
   status 0x0 (),

Otherwise the status would be 0x40. Section “Automatic Hardware Clock Synchronization by the Kernel” of hwclock’s manual has a very detailed explanation of how this status toggles “11 minute mode”, in which the kernel synchronizes RTC every 11 minutes. Another reference for RTC in Linux is https://www.kernel.org/doc/Documentation/rtc.txt

UTC Recommended

While it’s preferable to use local time zone on personal computers, you should set your servers to use UTC, especially if you run a global business, or if your local timezone has DST.

DST changes can be a real headache. Think of log processing and business analysis during DST change:

There could be a duplicated hour when DST ends.
The 23-hour and 25-hour days make day-to-day comparison non-sense.