[Dev] Winston / 1984 is feeling sad

[Luke Shumaker]

TL;DR: 1984 is having issues with some of their infrastructure, I've
removed one of their DNS servers that seems down from
winston.parabola.nu's network config.

The process:
 - Try to install something with pacman
 - Get timeouts
 - Discover that repomirror is taking 5-10 seconds to respond to each
   request
 - Log on to winston, notice that repomirror is leaking memory, and
   eating all the swap
 - Restart repomirror at winston
 - Still slow
 - Profiling says waiting for
   <https://www.parabola.nu/mirrors/status/tier/1/json/> is taking the
   time
 - Log on to proton, see nothing obvious, but things are slow
 - See most of the swap space belongs to uwsgi processes; restart all
   uwsgi services on proton, see better iowait / latency
 - Verify that the mirrors json URL now comes back quickly
 - repomirror still slow
 - Mirrors json URL is only slow *from winston*.
 - ping says connection between winston and proton is plenty fast, but
   took a while to say it
 - Verify that DNS lookup of www.parabola.nu is what is slow (winston
   uses 1984's DNS)
 - Try to log in to 1984 management panel to see what's up
 - Can't log in, get "Sorry, but it seems the backend server is
   feeling sad." from 1984's site.
 - See in winston's /etc/netctl/... that there are 2 DNS servers
   configured.  Ping them each.
 - One seems to be totally down, the other seems fine.
 - Comment out the down one, reload net configuration
 - Everything seems happy now
 - Notice pbot is offline, assume bad DNS killed him, restart him

-------------- next part --------------
A non-text attachment was scrubbed...
Name: sad.png
Type: image/png
Size: 45490 bytes
Desc: not available
URL: <https://lists.parabola.nu/pipermail/dev/attachments/20171117/dba2ae6c/attachment.png>
-------------- next part --------------

-- 
Happy hacking,
~ Luke Shumaker