[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Spacewalk-list] Problem with channel



OK. I manage to find a fix, so I share it ;)

The problem was the cache of the channel. It was in status "in
progress" when checking it by the web interface. Then I checked in
/var/log/rhn/rhn_taskomatic_daemon.log and saw a posgresql out of
memory error (log extract is below). So I figured out a
misconfiguration from me of postgres and changed it there :
/var/lib/pgsql/data/postgresql.conf

After that, I restarted spacewalk-service and got another error about
an out of memory error, but this time in the JVM. I searched the web,
and call a collegue which is a java expert. we fix it by editing
/var/log/rhn/rhn_taskomatic_daemon.log and changed these values:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=2048

# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=2048

My collegue also pointed out what is according to him (and without him
knowing spacewalk at all before calling him) a misconfiguration in the
java configuration of the catalina server:
-XX:MaxNewSize=256
So I removed it and checking with jstat showed better memory behavior
according to him.

So, the final fix for my problem was to increase the memory of the
taskomatic JVM.

FIRST ERROR (postgres)
#######################
INFO   | jvm 1    | 2014/11/20 09:27:32 | 2014-11-20 09:27:32,037
[Thread-51] INFO
com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating
new repository metadata for channel 'rhel6-x86_64'(sha1)
 22709 packages, 3014 errata
INFO   | jvm 1    | 2014/11/20 09:27:52 | 2014-11-20 09:27:52,741
[Thread-51] ERROR com.redhat.rhn.common.db.datasource.CachedStatement
- Error while processing cached statement sql: SELECT p.id, pn.name as
n
ame, pevr.epoch as epoch,
INFO   | jvm 1    | 2014/11/20 09:27:52 |          pevr.version as
version, pevr.release as release,
INFO   | jvm 1    | 2014/11/20 09:27:52 |          p.summary,
p.description, pa.label as arch_label,
INFO   | jvm 1    | 2014/11/20 09:27:52 |          p.build_time,
p.path, p.package_size, p.payload_size, p.installed_size,
INFO   | jvm 1    | 2014/11/20 09:27:52 |          p.copyright,
p.vendor, p.build_host, p.header_start, p.header_end,
INFO   | jvm 1    | 2014/11/20 09:27:52 |          srpm.name as
source_rpm, pg.name as package_group_name,
INFO   | jvm 1    | 2014/11/20 09:27:52 |          cs.checksum,
cs.checksum_type as checksum_type,
INFO   | jvm 1    | 2014/11/20 09:27:52 |          prd.primary_xml as
primary_xml, prd.filelist as filelist_xml, prd.other as other_xml
INFO   | jvm 1    | 2014/11/20 09:27:52 |     FROM
INFO   | jvm 1    | 2014/11/20 09:27:52 |          rhnChannelPackage cp,
INFO   | jvm 1    | 2014/11/20 09:27:52 |          rhnPackage p
INFO   | jvm 1    | 2014/11/20 09:27:52 |             LEFT OUTER JOIN
rhnPackageGroup pg ON p.package_group = pg.id
INFO   | jvm 1    | 2014/11/20 09:27:52 |             LEFT OUTER JOIN
rhnSourceRpm srpm  ON p.source_rpm_id = srpm.id
INFO   | jvm 1    | 2014/11/20 09:27:52 |             LEFT JOIN
rhnPackageRepodata prd ON prd.package_id = p.id,
INFO   | jvm 1    | 2014/11/20 09:27:52 |          rhnPackageName pn,
INFO   | jvm 1    | 2014/11/20 09:27:52 |          rhnPackageEvr pevr,
INFO   | jvm 1    | 2014/11/20 09:27:52 |          rhnPackageArch pa,
INFO   | jvm 1    | 2014/11/20 09:27:52 |          rhnChecksumView cs
INFO   | jvm 1    | 2014/11/20 09:27:52 |     WHERE
INFO   | jvm 1    | 2014/11/20 09:27:52 |          cp.package_id = p.id
INFO   | jvm 1    | 2014/11/20 09:27:52 |          AND p.name_id = pn.id
INFO   | jvm 1    | 2014/11/20 09:27:52 |          AND p.evr_id = pevr.id
INFO   | jvm 1    | 2014/11/20 09:27:52 |          AND p.package_arch_id = pa.id
INFO   | jvm 1    | 2014/11/20 09:27:52 |          AND cp.channel_id = ?
INFO   | jvm 1    | 2014/11/20 09:27:52 |          AND p.checksum_id = cs.id
INFO   | jvm 1    | 2014/11/20 09:27:52 |     ORDER by 1
INFO   | jvm 1    | 2014/11/20 09:27:52 |
com.redhat.rhn.common.db.WrappedSQLException: Ran out of memory
retrieving query results.
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.redhat.rhn.common.translation.SqlExceptionTranslator.postgreSqlException(SqlExceptionTranslator.java:54)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.redhat.rhn.common.translation.SqlExceptionTranslator.sqlException(SqlExceptionTranslator.java:44)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.redhat.rhn.common.db.NamedPreparedStatement.execute(NamedPreparedStatement.java:143)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:467)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:443)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:345)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:351)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:287)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.redhat.rhn.common.db.datasource.SelectMode.execute(SelectMode.java:110)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.redhat.rhn.manager.task.TaskManager.getChannelPackageDtos(TaskManager.java:52)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.redhat.rhn.taskomatic.task.repomd.RpmRepositoryWriter.writeRepomdFiles(RpmRepositoryWriter.java:170)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.redhat.rhn.taskomatic.task.repomd.ChannelRepodataWorker.run(ChannelRepodataWorker.java:104)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
java.lang.Thread.run(Thread.java:744)
INFO   | jvm 1    | 2014/11/20 09:27:52 | Caused by:
org.postgresql.util.PSQLException: Ran out of memory retrieving query
results.
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1777)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:360)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.execute(NewProxyPreparedStatement.java:362)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
com.redhat.rhn.common.db.NamedPreparedStatement.execute(NamedPreparedStatement.java:140)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       ... 11 more
INFO   | jvm 1    | 2014/11/20 09:27:52 | Caused by:
java.lang.OutOfMemoryError: Java heap space
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
org.postgresql.core.PGStream.ReceiveTupleV3(PGStream.java:349)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1774)
INFO   | jvm 1    | 2014/11/20 09:27:52 |       ... 17 more
INFO   | jvm 1    | 2014/11/20 09:27:52 | 2014-11-20 09:27:52,742
[Thread-51] ERROR com.redhat.rhn.taskomatic.task.ChannelRepodata -
com.redhat.rhn.common.db.WrappedSQLException: Ran out of memory
retrieving query results.
INFO   | jvm 1    | 2014/11/20 09:27:52 |
com.redhat.rhn.common.db.WrappedSQLException: Ran out of memory
retrieving query results.

SECOND ERROR - java out of memory
###################################
INFO   | jvm 1    | 2014/11/20 10:19:05 | 2014-11-20 10:19:05,284
[Thread-53] INFO
com.redhat.rhn.taskomatic.task.repomd.RepositoryWriter - Generating
new repository metadata for channel 'rhel6-x86_64'(sha1) 22709
packages, 3014 errata
INFO   | jvm 1    | 2014/11/20 10:19:37 | Exception in thread
"Thread-53" java.lang.OutOfMemoryError: GC overhead limit exceeded
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
java.lang.reflect.Method.copy(Method.java:151)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
java.lang.reflect.ReflectAccess.copyMethod(ReflectAccess.java:136)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
sun.reflect.ReflectionFactory.copyMethod(ReflectionFactory.java:300)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
java.lang.Class.copyMethods(Class.java:2852)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
java.lang.Class.getMethods(Class.java:1467)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
com.redhat.rhn.common.util.MethodUtil.callMethod(MethodUtil.java:118)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
com.redhat.rhn.common.db.datasource.CachedStatement.addToObject(CachedStatement.java:776)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
com.redhat.rhn.common.db.datasource.CachedStatement.processResultSet(CachedStatement.java:626)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:474)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:443)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:345)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:351)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
com.redhat.rhn.common.db.datasource.CachedStatement.execute(CachedStatement.java:287)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
com.redhat.rhn.common.db.datasource.SelectMode.execute(SelectMode.java:110)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
com.redhat.rhn.manager.task.TaskManager.getChannelPackageDtos(TaskManager.java:52)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
com.redhat.rhn.taskomatic.task.repomd.RpmRepositoryWriter.writeRepomdFiles(RpmRepositoryWriter.java:170)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
com.redhat.rhn.taskomatic.task.repomd.ChannelRepodataWorker.run(ChannelRepodataWorker.java:104)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
INFO   | jvm 1    | 2014/11/20 10:19:37 |       at
java.lang.Thread.run(Thread.java:744)

2014-11-20 9:25 GMT+01:00 Nicolas Michel <be nicolas michel gmail com>:
> Hello guys,
>
> I have a problem with a channel: the packages are not available from
> the server clients.
>
> [root my_hostname ~]# yum repolist
> Loaded plugins: product-id, rhnplugin, security, subscription-manager
> This system is not registered to Red Hat Subscription Management. You
> can use subscription-manager to register.
> This system is receiving updates from RHN Classic or Red Hat Satellite.
> repo id                                       repo name
>                              status
> epel-rhel6-x86_64                       EPEL-RHEL6-x86_64
>                 13,045
> rhel6-x86_64                              RHEL6-x86_64
>                        0
> spacewalk-client-rhel6-x86_64     Spacewalk-Client-RHEL6-x86_64              44
> repolist: 13,089
>
> As you can see, the channel rhel6-x86_64 has no packages associated to
> itself, from the point of view of the client.
>
> From the spacewalk web interface, I do see that there are 22709
> packages associated to rhel6-x86_64.
>
> Channel Name                                   Packages
> RHEL6-x86_64                                   22709
>      EPEL-RHEL6-x86_64                     13045
>      HP-SDR-rhel-x86_64                      375
>      Spacewalk-Client-RHEL6-x86_64     44
>
> The fact is : my channel rhel6-x86_64 is the parent of the others. Is
> it a problem?
>
> --
> Nicolas MICHEL



-- 
Nicolas MICHEL


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]