[Spacewalk-list] Spacewalk 1.7 w/ postgresql crashed

[Wolfgang Neudorfer]

Hi,

starting Saturday 17/11/2012 01:46, our Spacewalk server started to send out multiple mails per minute (probably on each connection attempt of a client?) like this:

-------------------------------------------------
RHN TRACEBACK from spacewalk1:

Exception reported from spacewalk1
Time: Sat Nov 17 01:45:30 2012
Exception type <class 'spacewalk.server.rhnSQL.sql_base.SQLConnectError'>
Request object information:
URI: /XMLRPC
Remote Host: 192.168.254.xxx
Server Name: spacewalk1:443
Headers passed in:
    Accept-Encoding: identity
    CONTENT_LENGTH: 2325
    CONTENT_TYPE: text/xml
    DOCUMENT_ROOT: /var/www/html
    GATEWAY_INTERFACE: CGI/1.1
    HTTPS: 1
    HTTP_ACCEPT_ENCODING: identity
    HTTP_HOST: spacewalk1
    HTTP_USER_AGENT: rhn.rpclib.py/$Revision$
    HTTP_X_CLIENT_VERSION: 1
    HTTP_X_INFO: RPC Processor (C) Red Hat, Inc (version $Revision$)
    HTTP_X_RHN_TRANSPORT_CAPABILITY: follow-redirects=3
    HTTP_X_TRANSPORT_INFO: Extended Capabilities Transport (C) Red Hat, Inc (version $Revision$)
    Host: tsasecspacewalk1.sec
    PATH_INFO: 
    QUERY_STRING: 
    REMOTE_ADDR: 192.168.254.xxx
    REMOTE_PORT: 59649
    REQUEST_METHOD: POST
    REQUEST_URI: /XMLRPC
    SCRIPT_FILENAME: /usr/share/rhn/wsgi/xmlrpc.py
    SCRIPT_NAME: /XMLRPC
    SCRIPT_URI: https://tsasecspacewalk1.sec/XMLRPC
    SCRIPT_URL: /XMLRPC
    SERVER_ADDR: 192.168.254.xxx
    SERVER_ADMIN: root at localhost
    SERVER_NAME: spacewalk1
    SERVER_PORT: 443
    SERVER_PROTOCOL: HTTP/1.1
    SERVER_SIGNATURE: <address>Apache Server at spacewalk1 Port 443</address>

    SERVER_SOFTWARE: Apache
    User-Agent: rhn.rpclib.py/$Revision$
    X-Client-Version: 1
    X-Info: RPC Processor (C) Red Hat, Inc (version $Revision$)
    X-RHN-Transport-Capability: follow-redirects=3
    X-Transport-Info: Extended Capabilities Transport (C) Red Hat, Inc (version $Revision$)
    mod_wsgi.application_group: tsasecspacewalk1.sec|/xmlrpc
    mod_wsgi.callable_object: application
    mod_wsgi.handler_script: 
    mod_wsgi.input_chunked: 0
    mod_wsgi.listener_host: 
    mod_wsgi.listener_port: 443
    mod_wsgi.process_group: 
    mod_wsgi.request_handler: wsgi-script
    mod_wsgi.script_reloading: 1
    mod_wsgi.version: (3, 2)
    wsgi.errors: <mod_wsgi.Log object at 0x7f8e4a83d370>
    wsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f8e4a83c300>
    wsgi.input: <mod_wsgi.Input object at 0x7f8e4a83d330>
    wsgi.multiprocess: True
    wsgi.multithread: False
    wsgi.run_once: False
    wsgi.url_scheme: https
    wsgi.version: (1, 1)
-------------------------------------------------

Apparently, something happend to the postgres server. In the log I see:

-------------------------------------------------
LOG:  server process (PID 31999) was terminated by signal 9: Killed
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

... (the last 2 lines appear multiple times)

FATAL:  the database system is in recovery mode
FATAL:  the database system is in recovery mode
FATAL:  the database system is in recovery mode
FATAL:  the database system is in recovery mode

... (this line apprears multiple times)
-------------------------------------------------

The harddisk was not full, also RAM was ok. I restarted the host and Spacewalk seems to be fine. I can login an all hosts are there.

Any hints? I am running Spacewalk 1.7 on CentOS x64 6.3 with PostgresSQL 8.4.13.

Thanks,

Wolfgang