• Reduce systemd timers coalescence in Antergos default setup


    Hello fine gentleman.

    As an introduction to the topic, here is the output of systemd timers on a fairly new Antergos setup:

    $ systemctl list-timers
    NEXT                         LEFT         LAST                         PASSED       UNIT                         
    Wed 2017-09-27 01:25:48 WAT  2min 8s left n/a                          n/a          systemd-tmpfiles-clean.timer
    Thu 2017-09-28 00:00:00 WAT  22h left     Wed 2017-09-27 00:00:38 WAT  1h 23min ago logrotate.timer              
    Thu 2017-09-28 00:00:00 WAT  22h left     Wed 2017-09-27 00:00:38 WAT  1h 23min ago man-db.timer                 
    Thu 2017-09-28 00:00:00 WAT  22h left     Wed 2017-09-27 00:00:38 WAT  1h 23min ago shadow.timer                 
    Thu 2017-09-28 00:00:00 WAT  22h left     Wed 2017-09-27 00:00:38 WAT  1h 23min ago updatedb.timer
    

    Notice the daily traffic at 00:00 hours. This does not go unnoticed on machines like mine with middle-range HDDs and modest CPUs. And every day at 0:00 I experience a few seconds of global system slowdown.

    Here are the default configurations for these timers:

    $ systemctl cat logrotate.timer man-db.timer shadow.timer updatedb.timer 
    # /usr/lib/systemd/system/logrotate.timer
    [Unit]
    Description=Daily rotation of log files
    Documentation=man:logrotate(8) man:logrotate.conf(5)
    
    [Timer]
    OnCalendar=daily
    AccuracySec=12h
    Persistent=true
    
    [Install]
    WantedBy=timers.target
    
    # /usr/lib/systemd/system/man-db.timer
    [Unit]
    Description=Daily man-db cache update
    
    [Timer]
    OnCalendar=daily
    AccuracySec=1d
    Persistent=true
    
    # /usr/lib/systemd/system/shadow.timer
    [Unit]
    Description=Daily verification of password and group files
    
    [Timer]
    OnCalendar=daily
    AccuracySec=12h
    Persistent=true
    
    # /usr/lib/systemd/system/updatedb.timer
    [Unit]
    Description=Daily locate database update
    
    [Timer]
    OnCalendar=daily
    AccuracySec=12h
    Persistent=true
    

    So, the problem is the combination of a calendar-based activation with a large AccuracySec value. What this means is that all these timers will always coalesce into the same moment in time within a 12 hour window (or even 1d on the case of man-db). So any two or more timers that would elapse within a 12 hour window of another will elapse at the same time and thus execute their respective service unit files.

    This situation is then worsened by the fact that the calendar time daily means the 00:00:00 of every day. So it is guaranteed all these services coalesce into a single point in time, causing the noticeable slowdown. And this is the reason list-timers always produces an output similar to the above.

    Suggestion:

    With certain type of servers, this is a good setup. But with desktops, which are the target audience of Antergos, we need to spread out these services a little so they go unnoticed by the user.

    To this effect, I would suggest replacing the AccuracySec directive with RandomizedDelaySec and set it to a value like 5m. Meaning randomize each of the aforementioned timers elapse period within a window of 5 minutes starting at 00:00:00.

    What do you think?

  • @Krugar said in Reduce systemd timers coalescence in Antergos default setup:

    Randomized

    I would vote for it!
    but i need to try first…

    [updates once a week] = [90% less problems]
    http://gofccyourself.com
    my-blog#k
    how to add system logs
    i3 GNOME

  • As an FYI, here’s how I set it up on my computer:

    I decided to override the perp timers:

    $ sudo -E systemctl edit shadow.timer
    $ sudo -E systemctl edit logrotate.timer
    $ sudo -E systemctl edit man-db.timer
    $ sudo -E systemctl edit updatedb.timer
    

    With these contents:

    [Timer]
    AccuracySec=1us
    RandomizedDelaySec=5m
    

    After reloading the systemd daemon, this is my new and friendlier list-timers output:

    $ sudo systemctl daemon-reload 
    $ systemctl list-timers
    NEXT                         LEFT          LAST                         PASSED    UNIT                         ACTIVATES
    Thu 2017-09-28 00:01:00 WAT  5h 9min left  Wed 2017-09-27 00:00:38 WAT  18h ago   logrotate.timer              logrotate
    Thu 2017-09-28 00:02:15 WAT  5h 10min left Wed 2017-09-27 00:00:38 WAT  18h ago   shadow.timer                 shadow.se
    Thu 2017-09-28 00:04:20 WAT  5h 12min left Wed 2017-09-27 00:00:38 WAT  18h ago   man-db.timer                 man-db.se
    Thu 2017-09-28 00:04:58 WAT  5h 13min left Wed 2017-09-27 00:00:38 WAT  18h ago   updatedb.timer               updatedb.
    Thu 2017-09-28 18:11:00 WAT  23h left      Wed 2017-09-27 18:11:00 WAT  40min ago systemd-tmpfiles-clean.timer
    
  • @Krugar said in Reduce systemd timers coalescence in Antergos default setup:

    systemctl list-timers

    NEXT                          LEFT         LAST                          PASSED             UNIT                         ACTIVATES
    Thu 2017-09-28 00:00:47 CEST  3h 3min left Wed 2017-09-27 00:00:42 CEST  20h ago            logrotate.timer              logrotate.service
    Thu 2017-09-28 00:01:09 CEST  3h 3min left Wed 2017-09-27 00:00:42 CEST  20h ago            shadow.timer                 shadow.service
    Thu 2017-09-28 00:02:31 CEST  3h 4min left Wed 2017-09-27 00:00:42 CEST  20h ago            updatedb.timer               updatedb.service
    Thu 2017-09-28 00:04:39 CEST  3h 6min left Wed 2017-09-27 00:00:42 CEST  20h ago            man-db.timer                 man-db.service
    Thu 2017-09-28 11:13:03 CEST  14h left     Wed 2017-09-27 11:13:03 CEST  9h ago             systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
    Sun 2017-10-01 00:00:00 CEST  3 days left  Fri 2017-09-01 00:00:34 CEST  3 weeks 5 days ago pamac-cleancache.timer       pamac-cleancache.service
    Mon 2017-10-02 00:00:00 CEST  4 days left  Mon 2017-09-25 00:00:56 CEST  2 days ago         fstrim.timer                 fstrim.service
    

    [updates once a week] = [90% less problems]
    http://gofccyourself.com
    my-blog#k
    how to add system logs
    i3 GNOME

  • I misread the manual page. It seems there is no need to mess with AccuracySec in the override file.

    [Timer]
    RandomizedDelaySec=5m
    

    That is enough.

    Problem with all this is that these services packages would have to be added to antergos repo. Or does Cnchi have the necessary facilities to script in these override.conf files during installation?

  • Ok. Here it is after today’s update, with new random elapsed time generated for the next day:

    $ systemctl list-timers
    NEXT                         LEFT     LAST                         PASSED       UNIT
    Thu 2017-09-28 23:43:24 WAT  23h left Wed 2017-09-27 23:43:24 WAT  26min ago    systemd-tmpfiles-clean.timer
    Fri 2017-09-29 00:00:11 WAT  23h left Thu 2017-09-28 00:04:24 WAT  5min ago     pkgfile-update.timer
    Fri 2017-09-29 00:00:22 WAT  23h left Thu 2017-09-28 00:02:58 WAT  6min ago     updatedb.timer
    Fri 2017-09-29 00:01:44 WAT  23h left Thu 2017-09-28 00:00:24 WAT  9min ago     man-db.timer
    Fri 2017-09-29 00:03:38 WAT  23h left Thu 2017-09-28 00:05:04 WAT  4min 47s ago logrotate.timer
    Fri 2017-09-29 00:04:02 WAT  23h left Thu 2017-09-28 00:04:24 WAT  5min ago     shadow.timer
    

    The experience was much different than before. When midnight came, I didn’t experience the system slowdown anymore and the whole period until the last timer elapsed at 00:05:04 was completely uneventful.

    You may noticed, I added the pkgfile update timer to crowd it a little more. Considering how well the system performed even with this addition, I will just let it stay there now.

  • Ok, so I’d like to make the proposal to the developers, but I don’t know where exactly. Looking at the github projects, I reckon Antergos/Cnchi maybe?

  • @Krugar said in Reduce systemd timers coalescence in Antergos default setup:

    Ok, so I’d like to make the proposal to the developers, but I don’t know where exactly. Looking at the github projects, I reckon Antergos/Cnchi maybe?

    should be the right place or antergos-iso as we both do not know whre this can be implemented

    [updates once a week] = [90% less problems]
    http://gofccyourself.com
    my-blog#k
    how to add system logs
    i3 GNOME

  • @Krugar
    I have not enough knowledge about your post’s subject, but i must say that your explanations seem very logic to my newbie mind :)

    Antergos (default OS) - WIN10 (abandoned)
    I3wm - Mate desktop
    AMD - A4 7300 Radeon graphics
    16 GB ram
    HD 1 TB
    Linux newbie since 06/2016

  • @fernandomaroto said in Reduce systemd timers coalescence in Antergos default setup:

    I have not enough knowledge about your post’s subject, but i must say that your explanations seem very logic to my newbie mind

    I must second that thought! I don’t quite understand everything going on in this htread, but it sounds like a good idea:grinning:.

    Keep trying, never give up. In the end, you will find that it was all worth it

    My ISO:
    https://rebornos.wordpress.com
    Linux Basics: https://linuxbasicssite.wordpress.com

  • Timers can be used as an alternative to cron (read #As a cron replacement). Timers have built-in support for calendar time events, monotonic time events, and can be run asynchronously

    they invoke commands repeating after a period of time.
    If the command invoke something that slow down your system like a databesupdate or tracker of files or removing big amount of data, it is a better idea to do not have this happen all at the same time.

    This is the idea here ;)

    [updates once a week] = [90% less problems]
    http://gofccyourself.com
    my-blog#k
    how to add system logs
    i3 GNOME

  • Ahhh… it’s all starting to make sense now…

    Keep trying, never give up. In the end, you will find that it was all worth it

    My ISO:
    https://rebornos.wordpress.com
    Linux Basics: https://linuxbasicssite.wordpress.com

  • I have a three part tutorial of Systemd pretty much finished – just doing some final touches – that I plan to add to Antergos wiki pretty soon. That will also help (or so I hope) to get a good understanding of the commands and concepts here. I actually noticed the problem discussed on this thread when I was writing the tutorial.

  • @Krugar said in Reduce systemd timers coalescence in Antergos default setup:

    I have a three part tutorial of Systemd pretty much finished – just doing some final touches – that I plan to add to Antergos wiki pretty soon

    Yay !!! I can’t wait to see it:grinning:.

    Keep trying, never give up. In the end, you will find that it was all worth it

    My ISO:
    https://rebornos.wordpress.com
    Linux Basics: https://linuxbasicssite.wordpress.com

  • Great! I just posted a response to it so that hopefully it can gain some more attention sooner ;). The more the merrier you know!

    Keep trying, never give up. In the end, you will find that it was all worth it

    My ISO:
    https://rebornos.wordpress.com
    Linux Basics: https://linuxbasicssite.wordpress.com

reduce2 timers2 systemd25 coalescence1 Posts 16Views 438
Log in to reply