From root at localhost.parabola.nu  Sun Sep 10 09:50:25 2017
From: root at localhost.parabola.nu (root at localhost.parabola.nu)
Date: Sun, 10 Sep 2017 08:50:25 -0000
Subject: [Maintenance] [Django] ERROR (internal IP): Internal Server Error:
 /mirrors/mirror
Message-ID: <20170910085025.1206.87350@proton.parabola.nu>

Internal Server Error: /mirrors/mirror
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 87, in get_response
    response = middleware_method(request)
  File "/usr/lib/python2.7/site-packages/django/middleware/common.py", line 82, in process_request
    "settings.") % (new_url[0], new_url[1]))
RuntimeError: You called this URL via POST, but the URL doesn't end in a slash and you have APPEND_SLASH set. Django can't redirect to the slash URL while maintaining POST data. Change your form to point to www.parabola.nu/mirrors/mirror/ (note the trailing slash), or set APPEND_SLASH=False in your Django settings.

Request repr(): 
<WSGIRequest
path:/mirrors/mirror,
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'CONTENT_LENGTH': '0',
 'CONTENT_TYPE': '',
 'DOCUMENT_ROOT': '/srv/http/www.parabola.nu',
 'HTTPS': 'on',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
 'HTTP_CONNECTION': 'keep-alive',
 'HTTP_CONTENT_LENGTH': '0',
 'HTTP_HOST': 'www.parabola.nu',
 'HTTP_USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 4.0.30319.17929)',
 'PATH_INFO': u'/mirrors/mirror',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_PORT': '2181',
 'REQUEST_METHOD': 'POST',
 'REQUEST_SCHEME': 'https',
 'REQUEST_URI': '/mirrors/mirror',
 u'SCRIPT_NAME': u'',
 'SERVER_NAME': 'www.parabola.nu',
 'SERVER_PORT': '443',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'uwsgi.node': 'proton.parabola.nu',
 'uwsgi.version': '2.0.15',
 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x7f73797ba390>,
 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
 'wsgi.input': <uwsgi._Input object at 0x7f7379229e10>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'https',
 'wsgi.version': (1, 0)}>

From root at localhost.parabola.nu  Sun Sep 10 16:01:11 2017
From: root at localhost.parabola.nu (root at localhost.parabola.nu)
Date: Sun, 10 Sep 2017 15:01:11 -0000
Subject: [Maintenance] [Django] ERROR (internal IP): Internal Server Error:
 /mirrors/mirror
Message-ID: <20170910150111.1206.80973@proton.parabola.nu>

Internal Server Error: /mirrors/mirror
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 87, in get_response
    response = middleware_method(request)
  File "/usr/lib/python2.7/site-packages/django/middleware/common.py", line 82, in process_request
    "settings.") % (new_url[0], new_url[1]))
RuntimeError: You called this URL via POST, but the URL doesn't end in a slash and you have APPEND_SLASH set. Django can't redirect to the slash URL while maintaining POST data. Change your form to point to www.parabola.nu/mirrors/mirror/ (note the trailing slash), or set APPEND_SLASH=False in your Django settings.

Request repr(): 
<WSGIRequest
path:/mirrors/mirror,
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'CONTENT_LENGTH': '0',
 'CONTENT_TYPE': '',
 'DOCUMENT_ROOT': '/srv/http/www.parabola.nu',
 'HTTPS': 'on',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
 'HTTP_CONNECTION': 'keep-alive',
 'HTTP_CONTENT_LENGTH': '0',
 'HTTP_HOST': 'www.parabola.nu',
 'HTTP_USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 4.0.30319.17929)',
 'PATH_INFO': u'/mirrors/mirror',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_PORT': '56612',
 'REQUEST_METHOD': 'POST',
 'REQUEST_SCHEME': 'https',
 'REQUEST_URI': '/mirrors/mirror',
 u'SCRIPT_NAME': u'',
 'SERVER_NAME': 'www.parabola.nu',
 'SERVER_PORT': '443',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'uwsgi.node': 'proton.parabola.nu',
 'uwsgi.version': '2.0.15',
 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x7f73797ba390>,
 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
 'wsgi.input': <uwsgi._Input object at 0x7f737975d948>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'https',
 'wsgi.version': (1, 0)}>

From lukeshu at lukeshu.com  Mon Sep 11 22:40:59 2017
From: lukeshu at lukeshu.com (Luke Shumaker)
Date: Mon, 11 Sep 2017 17:40:59 -0400
Subject: [Maintenance] nshd lockups
Message-ID: <87lglkvrpw.wl-lukeshu@lukeshu.com>

Of the last 4 outages, one was caused by the hardware being physically
offline.  The other 3 were:

  | # | Started                 | Resolved                | Host                |
  |---|-------------------------|-------------------------|---------------------|
  | 1 | 2017-08-13 23:00:09 UTC | 2017-08-16 16:58:04 UTC | proton.parabola.nu  |
  | 2 | 2017-08-29 23:04:31 UTC | 2017-08-31 02:57:05 UTC | winston.parabola.nu |
  | 3 | 2017-09-11 18:29:35 UTC | 2017-09-11 19:36:27 UTC | winston.parabola.nu |

After resolving incident #2, I said that I would investigate it more
that Friday (so this message is a bit late).  I have concluded that
both that outage, and the other two listed, were caused by nshd
lockups, causing NSS lookups to block forever.

One of the symptoms of the nshd lockup is that ssh logins are
impossible, even on the emergency@ user.

  While responding to incident #2 I noted that it was weird that I was
  able to get an ssh login on repo at .  That succeeded because it was a
  pre-existing background connection created by librerelease
  HOOKPRERELEASE at 2017-07-28 16:33:38 UTC.  New connections are
  impossible.

This causes problems for us developers, but is generally invisible to
users (a good thing), but that means that it may be a bit before the
issue is noticed (explaining the long outage times).

On winston, we can use the 1984 VPS control panel to reboot it.
However, on proton, without access to the emergency@ user, we have to
get in touch with n1md4 to have it reboot.

I believe what was happening is that a client hangs up (which could be
literally any process on the system getting killed) in the middle of
an NSS lookup with nshd, and nshd never completes its response, never
freeing the lock associated with the request, which creates a deadlock
the next time it receives SIGHUP to reload; the reload routine will
never acquire the lock, but it will block incoming requests from
getting it.  Boom.

In response, I have released version 20170908 of nshd (the
parabola-hackers-nshd package) that I believe resolves the deadlock.
It added per-request limits; most importantly read and write timeouts
(and maximum request size, fixing a different local-user DoS attack
https://labs.parabola.nu/issues/1068).

Despite being released on the 8th, the new nshd version had not yet
been deployed to the servers in time to avoid incident #3 today.  In
response to #3, I have updated nshd (and everything else; -Syu) on
both servers.

Next steps:

 - Perhaps have parabola-hackers load the user information in to an
   SQL database, and have a standard SQL nss/pam module talk to that;
   get our custom code out of the hot-path.
   https://labs.parabola.nu/issues/1465

 - Find a way to ensure that nshd can't block emergency at .  The user
   lookup succeeds, but when ssh'ing to emergency@, it needs to look
   up emergency's group membership, which *does* hit nshd.  Perhaps
   add a special case in the Group_ByMember handler that before even
   grabbing the lock returns if the user us "emergency".

-- 
Happy hacking,
~ Luke Shumaker

From root at localhost.parabola.nu  Sun Sep 17 16:02:04 2017
From: root at localhost.parabola.nu (root at localhost.parabola.nu)
Date: Sun, 17 Sep 2017 15:02:04 -0000
Subject: [Maintenance] [Django] ERROR (internal IP): Internal Server Error:
 /mirrors/mirror
Message-ID: <20170917150204.1055.30713@proton.parabola.nu>

Internal Server Error: /mirrors/mirror
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 87, in get_response
    response = middleware_method(request)
  File "/usr/lib/python2.7/site-packages/django/middleware/common.py", line 82, in process_request
    "settings.") % (new_url[0], new_url[1]))
RuntimeError: You called this URL via POST, but the URL doesn't end in a slash and you have APPEND_SLASH set. Django can't redirect to the slash URL while maintaining POST data. Change your form to point to www.parabola.nu/mirrors/mirror/ (note the trailing slash), or set APPEND_SLASH=False in your Django settings.

Request repr(): 
<WSGIRequest
path:/mirrors/mirror,
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'CONTENT_LENGTH': '0',
 'CONTENT_TYPE': '',
 'DOCUMENT_ROOT': '/srv/http/www.parabola.nu',
 'HTTPS': 'on',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
 'HTTP_CONNECTION': 'keep-alive',
 'HTTP_CONTENT_LENGTH': '0',
 'HTTP_HOST': 'www.parabola.nu',
 'HTTP_USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 4.0.30319.17929)',
 'PATH_INFO': u'/mirrors/mirror',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_PORT': '54920',
 'REQUEST_METHOD': 'POST',
 'REQUEST_SCHEME': 'https',
 'REQUEST_URI': '/mirrors/mirror',
 u'SCRIPT_NAME': u'',
 'SERVER_NAME': 'www.parabola.nu',
 'SERVER_PORT': '443',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'uwsgi.node': 'proton.parabola.nu',
 'uwsgi.version': '2.0.15',
 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x7fdbdcf42390>,
 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
 'wsgi.input': <uwsgi._Input object at 0x7fdbdc96dba0>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'https',
 'wsgi.version': (1, 0)}>

From root at localhost.parabola.nu  Sun Sep 17 18:52:08 2017
From: root at localhost.parabola.nu (root at localhost.parabola.nu)
Date: Sun, 17 Sep 2017 17:52:08 -0000
Subject: [Maintenance] [Django] ERROR (internal IP): Internal Server Error:
 /mirrors/mirror
Message-ID: <20170917175208.1055.83394@proton.parabola.nu>

Internal Server Error: /mirrors/mirror
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 87, in get_response
    response = middleware_method(request)
  File "/usr/lib/python2.7/site-packages/django/middleware/common.py", line 82, in process_request
    "settings.") % (new_url[0], new_url[1]))
RuntimeError: You called this URL via POST, but the URL doesn't end in a slash and you have APPEND_SLASH set. Django can't redirect to the slash URL while maintaining POST data. Change your form to point to www.parabola.nu/mirrors/mirror/ (note the trailing slash), or set APPEND_SLASH=False in your Django settings.

Request repr(): 
<WSGIRequest
path:/mirrors/mirror,
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'CONTENT_LENGTH': '0',
 'CONTENT_TYPE': '',
 'DOCUMENT_ROOT': '/srv/http/www.parabola.nu',
 'HTTPS': 'on',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
 'HTTP_CONNECTION': 'keep-alive',
 'HTTP_CONTENT_LENGTH': '0',
 'HTTP_HOST': 'www.parabola.nu',
 'HTTP_USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 4.0.30319.17929)',
 'PATH_INFO': u'/mirrors/mirror',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_PORT': '52019',
 'REQUEST_METHOD': 'POST',
 'REQUEST_SCHEME': 'https',
 'REQUEST_URI': '/mirrors/mirror',
 u'SCRIPT_NAME': u'',
 'SERVER_NAME': 'www.parabola.nu',
 'SERVER_PORT': '443',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'uwsgi.node': 'proton.parabola.nu',
 'uwsgi.version': '2.0.15',
 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x7fdbdcf42390>,
 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
 'wsgi.input': <uwsgi._Input object at 0x7fdbdc8e47b0>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'https',
 'wsgi.version': (1, 0)}>

From root at localhost.parabola.nu  Sun Sep 17 20:22:31 2017
From: root at localhost.parabola.nu (root at localhost.parabola.nu)
Date: Sun, 17 Sep 2017 19:22:31 -0000
Subject: [Maintenance] [Django] ERROR (internal IP): Internal Server Error:
 /mirrors/mirror
Message-ID: <20170917192231.1052.78431@proton.parabola.nu>

Internal Server Error: /mirrors/mirror
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 87, in get_response
    response = middleware_method(request)
  File "/usr/lib/python2.7/site-packages/django/middleware/common.py", line 82, in process_request
    "settings.") % (new_url[0], new_url[1]))
RuntimeError: You called this URL via POST, but the URL doesn't end in a slash and you have APPEND_SLASH set. Django can't redirect to the slash URL while maintaining POST data. Change your form to point to www.parabola.nu/mirrors/mirror/ (note the trailing slash), or set APPEND_SLASH=False in your Django settings.

Request repr(): 
<WSGIRequest
path:/mirrors/mirror,
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'CONTENT_LENGTH': '0',
 'CONTENT_TYPE': '',
 'DOCUMENT_ROOT': '/srv/http/www.parabola.nu',
 'HTTPS': 'on',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
 'HTTP_CONNECTION': 'keep-alive',
 'HTTP_CONTENT_LENGTH': '0',
 'HTTP_HOST': 'www.parabola.nu',
 'HTTP_USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 4.0.30319.17929)',
 'PATH_INFO': u'/mirrors/mirror',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_PORT': '53074',
 'REQUEST_METHOD': 'POST',
 'REQUEST_SCHEME': 'https',
 'REQUEST_URI': '/mirrors/mirror',
 u'SCRIPT_NAME': u'',
 'SERVER_NAME': 'www.parabola.nu',
 'SERVER_PORT': '443',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'uwsgi.node': 'proton.parabola.nu',
 'uwsgi.version': '2.0.15',
 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x7fdbdcf42390>,
 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
 'wsgi.input': <uwsgi._Input object at 0x7fdbdc610cf0>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'https',
 'wsgi.version': (1, 0)}>

From root at localhost.parabola.nu  Mon Sep 18 13:35:31 2017
From: root at localhost.parabola.nu (root at localhost.parabola.nu)
Date: Mon, 18 Sep 2017 12:35:31 -0000
Subject: [Maintenance] [Django] ERROR (internal IP): Internal Server Error:
 /packages/libre-testing/i686/glibc/files/
Message-ID: <20170918123531.1052.31869@proton.parabola.nu>

Internal Server Error: /packages/libre-testing/i686/glibc/files/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "./packages/views/display.py", line 190, in files
    return render(request, template, context)
  File "/usr/lib/python2.7/site-packages/django/shortcuts.py", line 50, in render
    return HttpResponse(loader.render_to_string(*args, **kwargs),
  File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 178, in render_to_string
    return t.render(context_instance)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 148, in render
    return self._render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 142, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/site-packages/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 126, in render
    return compiled_parent._render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 142, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/site-packages/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 65, in render
    result = block.nodelist.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/site-packages/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 150, in render
    return template.render(context)
  File "/usr/lib/python2.7/site-packages/django_jinja/base.py", line 90, in render
    return super(Template, self).render(new_context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/srv/http/www.parabola.nu/web/templates/packages/files_list.html.jinja", line 1, in top-level template code
    {% if pkg.last_update > pkg.files_last_update %}
TypeError: can't compare datetime.datetime to NoneType

Request repr(): 
<WSGIRequest
path:/packages/libre-testing/i686/glibc/files/,
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'CONTENT_LENGTH': '',
 'CONTENT_TYPE': '',
 u'CSRF_COOKIE': u'iHG2tn8kbTW9Q6vO05L6r4jfbhWvgpY4',
 'DOCUMENT_ROOT': '/srv/http/www.parabola.nu',
 'HTTPS': 'on',
 'HTTP_ACCEPT': 'text/html, text/*, application/*, image/*, message/*',
 'HTTP_ACCEPT_ENCODING': 'gzip',
 'HTTP_CONNECTION': 'close',
 'HTTP_HOST': 'www.parabola.nu',
 'HTTP_USER_AGENT': 'ltx71 - (http://ltx71.com/)',
 'PATH_INFO': u'/packages/libre-testing/i686/glibc/files/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_PORT': '57564',
 'REQUEST_METHOD': 'GET',
 'REQUEST_SCHEME': 'https',
 'REQUEST_URI': '/packages/libre-testing/i686/glibc/files/',
 u'SCRIPT_NAME': u'',
 'SERVER_NAME': 'www.parabola.nu',
 'SERVER_PORT': '443',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'uwsgi.node': 'proton.parabola.nu',
 'uwsgi.version': '2.0.15',
 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x7fdbdcf42390>,
 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
 'wsgi.input': <uwsgi._Input object at 0x7fdbdc949d50>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'https',
 'wsgi.version': (1, 0)}>

From root at localhost.parabola.nu  Sat Sep 23 21:54:52 2017
From: root at localhost.parabola.nu (root at localhost.parabola.nu)
Date: Sat, 23 Sep 2017 20:54:52 -0000
Subject: [Maintenance] [Django] ERROR (internal IP): Internal Server Error:
 /packages/libre-testing/i686/binutils/files/
Message-ID: <20170923205452.1052.73189@proton.parabola.nu>

Internal Server Error: /packages/libre-testing/i686/binutils/files/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "./packages/views/display.py", line 190, in files
    return render(request, template, context)
  File "/usr/lib/python2.7/site-packages/django/shortcuts.py", line 50, in render
    return HttpResponse(loader.render_to_string(*args, **kwargs),
  File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 178, in render_to_string
    return t.render(context_instance)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 148, in render
    return self._render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 142, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/site-packages/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 126, in render
    return compiled_parent._render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 142, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/site-packages/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 65, in render
    result = block.nodelist.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/site-packages/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 150, in render
    return template.render(context)
  File "/usr/lib/python2.7/site-packages/django_jinja/base.py", line 90, in render
    return super(Template, self).render(new_context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/srv/http/www.parabola.nu/web/templates/packages/files_list.html.jinja", line 1, in top-level template code
    {% if pkg.last_update > pkg.files_last_update %}
TypeError: can't compare datetime.datetime to NoneType

Request repr(): 
<WSGIRequest
path:/packages/libre-testing/i686/binutils/files/,
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'CONTENT_LENGTH': '',
 'CONTENT_TYPE': '',
 u'CSRF_COOKIE': u'vDYTuseZq7wD6qN0O2rDMQ6amN81rSQe',
 'DOCUMENT_ROOT': '/srv/http/www.parabola.nu',
 'HTTPS': 'on',
 'HTTP_ACCEPT': 'text/html, text/*, application/*, image/*, message/*',
 'HTTP_ACCEPT_ENCODING': 'gzip',
 'HTTP_CONNECTION': 'close',
 'HTTP_HOST': 'www.parabola.nu',
 'HTTP_USER_AGENT': 'ltx71 - (http://ltx71.com/)',
 'PATH_INFO': u'/packages/libre-testing/i686/binutils/files/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_PORT': '40096',
 'REQUEST_METHOD': 'GET',
 'REQUEST_SCHEME': 'https',
 'REQUEST_URI': '/packages/libre-testing/i686/binutils/files/',
 u'SCRIPT_NAME': u'',
 'SERVER_NAME': 'www.parabola.nu',
 'SERVER_PORT': '443',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'uwsgi.node': 'proton.parabola.nu',
 'uwsgi.version': '2.0.15',
 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x7fdbdcf42390>,
 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
 'wsgi.input': <uwsgi._Input object at 0x7fdbdc949030>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'https',
 'wsgi.version': (1, 0)}>

From root at localhost.parabola.nu  Sun Sep 24 02:23:52 2017
From: root at localhost.parabola.nu (root at localhost.parabola.nu)
Date: Sun, 24 Sep 2017 01:23:52 -0000
Subject: [Maintenance] [Django] ERROR (internal IP): Internal Server Error:
 /packages/extra/any/kde-meta-kdeedu/files/
Message-ID: <20170924012352.1052.37088@proton.parabola.nu>

Internal Server Error: /packages/extra/any/kde-meta-kdeedu/files/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "./packages/views/display.py", line 190, in files
    return render(request, template, context)
  File "/usr/lib/python2.7/site-packages/django/shortcuts.py", line 50, in render
    return HttpResponse(loader.render_to_string(*args, **kwargs),
  File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 178, in render_to_string
    return t.render(context_instance)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 148, in render
    return self._render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 142, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/site-packages/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 126, in render
    return compiled_parent._render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 142, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/site-packages/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 65, in render
    result = block.nodelist.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/site-packages/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 150, in render
    return template.render(context)
  File "/usr/lib/python2.7/site-packages/django_jinja/base.py", line 90, in render
    return super(Template, self).render(new_context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/srv/http/www.parabola.nu/web/templates/packages/files_list.html.jinja", line 1, in top-level template code
    {% if pkg.last_update > pkg.files_last_update %}
TypeError: can't compare datetime.datetime to NoneType

Request repr(): 
<WSGIRequest
path:/packages/extra/any/kde-meta-kdeedu/files/,
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'CONTENT_LENGTH': '',
 'CONTENT_TYPE': '',
 u'CSRF_COOKIE': u'2a3huVjp9E461K1sxOrJ1XzcwVjIy66b',
 'DOCUMENT_ROOT': '/srv/http/www.parabola.nu',
 'HTTPS': 'on',
 'HTTP_ACCEPT': 'text/html, text/*, application/*, image/*, message/*',
 'HTTP_ACCEPT_ENCODING': 'gzip',
 'HTTP_CONNECTION': 'close',
 'HTTP_HOST': 'www.parabola.nu',
 'HTTP_USER_AGENT': 'ltx71 - (http://ltx71.com/)',
 'PATH_INFO': u'/packages/extra/any/kde-meta-kdeedu/files/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_PORT': '40476',
 'REQUEST_METHOD': 'GET',
 'REQUEST_SCHEME': 'https',
 'REQUEST_URI': '/packages/extra/any/kde-meta-kdeedu/files/',
 u'SCRIPT_NAME': u'',
 'SERVER_NAME': 'www.parabola.nu',
 'SERVER_PORT': '443',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'uwsgi.node': 'proton.parabola.nu',
 'uwsgi.version': '2.0.15',
 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x7fdbdcf42390>,
 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
 'wsgi.input': <uwsgi._Input object at 0x7fdbdca9bac8>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'https',
 'wsgi.version': (1, 0)}>

From root at localhost.parabola.nu  Thu Sep 28 22:37:46 2017
From: root at localhost.parabola.nu (root at localhost.parabola.nu)
Date: Thu, 28 Sep 2017 21:37:46 -0000
Subject: [Maintenance] [Django] ERROR (internal IP): Internal Server Error:
 /packages/libre/i686/libsystemd/files/
Message-ID: <20170928213746.1053.94404@proton.parabola.nu>

Internal Server Error: /packages/libre/i686/libsystemd/files/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "./packages/views/display.py", line 190, in files
    return render(request, template, context)
  File "/usr/lib/python2.7/site-packages/django/shortcuts.py", line 50, in render
    return HttpResponse(loader.render_to_string(*args, **kwargs),
  File "/usr/lib/python2.7/site-packages/django/template/loader.py", line 178, in render_to_string
    return t.render(context_instance)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 148, in render
    return self._render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 142, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/site-packages/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 126, in render
    return compiled_parent._render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 142, in _render
    return self.nodelist.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/site-packages/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 65, in render
    result = block.nodelist.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/usr/lib/python2.7/site-packages/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/usr/lib/python2.7/site-packages/django/template/loader_tags.py", line 150, in render
    return template.render(context)
  File "/usr/lib/python2.7/site-packages/django_jinja/base.py", line 90, in render
    return super(Template, self).render(new_context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/srv/http/www.parabola.nu/web/templates/packages/files_list.html.jinja", line 1, in top-level template code
    {% if pkg.last_update > pkg.files_last_update %}
TypeError: can't compare datetime.datetime to NoneType

Request repr(): 
<WSGIRequest
path:/packages/libre/i686/libsystemd/files/,
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'CONTENT_LENGTH': '',
 'CONTENT_TYPE': '',
 u'CSRF_COOKIE': u'Wi5YATA7m1yWisygTzDUOE5LNSNyDBmt',
 'DOCUMENT_ROOT': '/srv/http/www.parabola.nu',
 'HTTPS': 'on',
 'HTTP_ACCEPT': 'text/html, text/*, application/*, image/*, message/*',
 'HTTP_ACCEPT_ENCODING': 'gzip',
 'HTTP_CONNECTION': 'close',
 'HTTP_HOST': 'www.parabola.nu',
 'HTTP_USER_AGENT': 'ltx71 - (http://ltx71.com/)',
 'PATH_INFO': u'/packages/libre/i686/libsystemd/files/',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_PORT': '35070',
 'REQUEST_METHOD': 'GET',
 'REQUEST_SCHEME': 'https',
 'REQUEST_URI': '/packages/libre/i686/libsystemd/files/',
 u'SCRIPT_NAME': u'',
 'SERVER_NAME': 'www.parabola.nu',
 'SERVER_PORT': '443',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'uwsgi.node': 'proton.parabola.nu',
 'uwsgi.version': '2.0.15',
 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x7fdbdcf42390>,
 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>,
 'wsgi.input': <uwsgi._Input object at 0x7fdbdca90d08>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'https',
 'wsgi.version': (1, 0)}>