This post is about how misconfigured time can affect the Linux operating system and lead to unexpected results. Also I’ll share some thoughts on how to get it right. In this specific example, the Linux distribution is CentOS 7.
Components Involved
- RTC – real time clock. It is sometimes called hardware clock, can be set in BIOS setup screen or from the OS.
- System clock – the software clock maintained by OS kernel.
- chrony – successor to ntpd that keeps the system clock in sync.
- memcached – software that keeps its own time.
What Went Wrong
We rebooted several memcached servers to replace their motherboards. When they were back online, memcached clients could successfully set keys in the nodes, but were not able to get anything out of them.
How it Went Wrong
stats of those memcached nodes showed that cmd_set
and get_misses
steadily increased, but get_hits
and bytes
always kept the same.
That made me look into the usage pattern of the memcached cluster. Luckily that was a new cluster and the only usage of it was to put in keys that expire in 10 minutes.
Intricacies of Memcached
Memcached is a little special in that it only reads time from system clock once on startup, and maintains its own clock thereafter.
And memcached’s protocol allows you to set the expiration time in seconds (up to 30 days) or as a Unix timestamp – a single field that could have multiple meanings. That seems to be an anti-pattern to today’s design principles. However, this multi-purpose field is directly supported by spymemcached’s implementation.
Then I took a quick peek at the clock that memcached secretly holds:
date -d @`echo stats | nc {memcached_address} {memcached_port} | grep "STAT time " | cut -d" " -f 3`
It showed that it’s 1 hour faster than system lock. We used a wrapper around spymemcached and to eliminate the ambiguity of expiration time, we made set()
method accept only java.util.Date
. So when the client set a key that would expire in 10 minutes, the server thought it had expired 50 minutes ago and evicted it immediately.
Messy Time
The error in memcached’s clock could be caused by error in system clock, before chronyd made it synchronized with time servers. How did system clock went wrong? Here’s the relation between hardware clock (or real time clock, RTC) and software clock:
The Linux kernel keeps track of time independently from the hardware clock. During the boot, Linux sets its own clock to the same time as the hardware clock. After this, both clocks run independently. Linux maintains its own clock because looking at the hardware is slow and complicated.
https://www.tldp.org/LDP/sag/html/hw-sw-clocks.html
So the hardware clock must have gone wrong. Running timedatectl
revealed that. Wait – I had rtcsync
directive in /etc/chrony.conf
. The hardware clock should have been corrected, shouldn’t it? timedatectl
warned at the end:
Warning: The system is configured to read the RTC time in the local time zone.
This mode can not be fully supported. It will create various problems
with time zone changes and daylight saving time adjustments. The RTC
time is never updated, it relies on external facilities to maintain it.
If at all possible, use RTC in UTC by calling
'timedatectl set-local-rtc 0'.
Now I must confess that the servers were configured not in UTC, but in a local time zone. To make it worse, the local time zone is a DST time zone.
As timedatectl
warned, we configured the system to read RTC as local time. RTC itself is like an old-style wall clock and is not timezone-aware. The kernel does calculations as needed when it reads or writes the RTC.
The Fix
So the fix was as simple as running what it had suggested:
timedatectl set-local-rtc 0
In my experience a reboot is needed to make it effective. Note that on this reboot, memcached can still get a wrong time from system clock. Rebooting memcached after time is synchronized can be a fix.
More Notes
rtconutc in chrony.conf
As far as I know (briefly looked into chrony’s source) This directive only have effect when rtcautotrim
is enabled, or when trimrtc
is run from chronyc
.
How rtcsync works
rtcsync
directive only lets chronyd set a status flag when invoking system call to adjust system clock. The effect can be seen in the output of timedatectl
as:
NTP synchronized: yes
Or in the output of ntptime
as:
ntp_adjtime() returns code 0 (OK)
...
status 0x0 (),
Otherwise the status would be 0x40
. Section “Automatic Hardware Clock Synchronization by the Kernel” of hwclock’s manual has a very detailed explanation of how this status toggles “11 minute mode”, in which the kernel synchronizes RTC every 11 minutes. Another reference for RTC in Linux is https://www.kernel.org/doc/Documentation/rtc.txt
UTC Recommended
While it’s preferable to use local time zone on personal computers, you should set your servers to use UTC, especially if you run a global business, or if your local timezone has DST.
DST changes can be a real headache. Think of log processing and business analysis during DST change:
- There could be a duplicated hour when DST ends.
- The 23-hour and 25-hour days make day-to-day comparison non-sense.