[Spacewalk-list] can't log in to web gui or run spacecmd

[Robin Beardsley]

Today I tried to log in to my SW server and could not get in.  I also tried to get in via spacecmd and could not get in.
This is from the  rhn_taskomatic_daemon.log

INFO   | jvm 5    | 2017/08/31 12:43:15 | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
INFO   | jvm 5    | 2017/08/31 12:43:15 |   Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.
INFO   | jvm 5    | 2017/08/31 12:43:15 |
ERROR  | wrapper  | 2017/08/31 12:43:44 | Startup failed: Timed out waiting for signal from JVM.
ERROR  | wrapper  | 2017/08/31 12:43:45 | JVM did not exit on request, terminated
INFO   | wrapper  | 2017/08/31 12:43:45 | JVM exited on its own while waiting to kill the application.
STATUS | wrapper  | 2017/08/31 12:43:45 | JVM exited in response to signal SIGKILL (9).
FATAL  | wrapper  | 2017/08/31 12:43:45 | There were 5 failed launches in a row, each lasting less than 300 seconds.  Giving up.
FATAL  | wrapper  | 2017/08/31 12:43:45 |   There may be a configuration problem: please check the logs.
STATUS | wrapper  | 2017/08/31 12:43:45 | <-- Wrapper Stopped


[root at oelsandbox rhn]# spacecmd
Welcome to spacecmd, a command-line interface to Spacewalk.

Type: 'help' for a list of commands
      'help <cmd>' for command-specific help
      'quit' to quit

Traceback (most recent call last):
  File "/bin/spacecmd", line 166, in <module>
    if not shell.do_login(''):
  File "/usr/lib/python2.7/site-packages/spacecmd/misc.py", line 284, in do_login
    self.api_version = self.client.api.getVersion()
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1321, in single_request
    response.msg,
xmlrpclib.ProtocolError: <ProtocolError for oelsandbox.nedelta.com/rpc/api: 500 Internal Server Error>

Everything has been running fine up until today.  I did a yum security update yesterday and the only update was a kernel update.  I rebooted the server.
All spacewalk services were restarted and they all seem to be running.

Tail of up2date log


[Wed Aug 30 10:52:31 2017] up2date Updating package profile
[Wed Aug 30 14:52:30 2017] up2date logging into up2date server
[Wed Aug 30 14:52:30 2017] up2date successfully retrieved authentication token from up2date server
[Wed Aug 30 18:52:30 2017] up2date logging into up2date server
[Wed Aug 30 18:52:30 2017] up2date successfully retrieved authentication token from up2date server
[Wed Aug 30 22:52:31 2017] up2date logging into up2date server
[Wed Aug 30 22:52:31 2017] up2date successfully retrieved authentication token from up2date server
[Thu Aug 31 02:52:31 2017] up2date logging into up2date server
[Thu Aug 31 02:52:31 2017] up2date successfully retrieved authentication token from up2date server
[Thu Aug 31 06:52:31 2017] up2date logging into up2date server
[Thu Aug 31 06:52:31 2017] up2date successfully retrieved authentication token from up2date server
[Thu Aug 31 12:32:52 2017] up2date logging into up2date server
[Thu Aug 31 12:32:53 2017] up2date A protocol error occurred: Internal Server Error , attempt #1,
[Thu Aug 31 12:35:25 2017] up2date logging into up2date server
[Thu Aug 31 12:35:32 2017] up2date A protocol error occurred: Internal Server Error , attempt #1,
[Thu Aug 31 12:35:48 2017] up2date logging into up2date server
[Thu Aug 31 12:36:18 2017] up2date A protocol error occurred: Internal Server Error , attempt #1,
[Thu Aug 31 12:48:08 2017] up2date logging into up2date server
[Thu Aug 31 12:48:38 2017] up2date A protocol error occurred: Internal Server Error , attempt #1,
[Thu Aug 31 12:55:08 2017] up2date logging into up2date server
[Thu Aug 31 12:55:35 2017] up2date A protocol error occurred: Internal Server Error , attempt #1,

[root at oelsandbox log]# uname -r
3.8.13-118.19.4.el7uek.x86_64
[root at oelsandbox log]# uptrack-uname -r
3.8.13-118.19.4.el7uek.x86_64

Does anyone have any idea what might be the cause of this?

Thank you





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20170831/919b38c7/attachment.htm>