Oracle/Red Hat/NetApp

[Stephen Carville]

Troy Knabe wrote:
> We have several Oracle instances 10.2 (some RAC, some not, but all are
> having the same issue) all running on Red Hat 4.5.  The binaries and data
> files are all on the NetApp and mounted via NFS.
> 
>  
> 
> Sporadically when we shut the database down, we are unable to restart it.
> 
> 
>  
> 
>  
> 
> /u1/app/oracle/admin/prodrac/bdump/prodrac1_dbw0_10023.trc
> 
>  
> 
> Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit
> Production
> 
> With the Partitioning, Real Application Clusters, OLAP and Data Mining
> options
> 
> ORACLE_HOME = /u1/app/oracle/product/10.2.0
> 
> System name:   Linux
> 
> Node name:       orarac1
> 
> Release:               2.6.9-55.0.2.ELsmp
> 
> Version:               #1 SMP Tue Jun 12 17:58:20 EDT 2007
> 
> Machine:             x86_64
> 
> Instance name: prodrac1
> 
> Redo thread mounted by this instance: 1
> 
> Oracle process number: 8
> 
> Unix process pid: 10023, image: oracle at orarac1 (DBW0)
> 
>  
> 
> *** SERVICE NAME:() 2007-08-18 02:05:27.878
> 
> *** SESSION ID:(1194.1) 2007-08-18 02:05:27.878
> 
> ORA-01157: cannot identify/lock data file 1 - see DBWR trace file
> 
> ORA-01110: data file 1: '/u1/oradata/prod/system/system01.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 2 - see DBWR trace file
> 
> ORA-01110: data file 2: '/u1/oradata/prod/temp/undotbs01.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 3 - see DBWR trace file
> 
> ORA-01110: data file 3: '/u1/oradata/prod/system/sysaux01.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 4 - see DBWR trace file
> 
> ORA-01110: data file 4: '/u1/oradata/prod/table/users01.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 5 - see DBWR trace file
> 
> ORA-01110: data file 5: '/u1/oradata/prod/temp/undotbs02.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 6 - see DBWR trace file
> 
> ORA-01110: data file 6: '/u1/oradata/prod/temp/undotbs03.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 7 - see DBWR trace file
> 
> ORA-01110: data file 7: '/u1/oradata/prod/table/esis_codes_data01.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 8 - see DBWR trace file
> 
> ORA-01110: data file 8: '/u1/oradata/prod/index/esis_codes_indx01.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 9 - see DBWR trace file
> 
> ORA-01110: data file 9: '/u1/oradata/prod/index/esis_large_indx.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 10 - see DBWR trace file
> 
> ORA-01110: data file 10: '/u1/oradata/prod/index/esis_temp_indx.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 11 - see DBWR trace file
> 
> ORA-01110: data file 11: '/u1/oradata/prod/index/esis_indx.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 12 - see DBWR trace file
> 
> ORA-01110: data file 12: '/u1/oradata/prod/table/school_connects.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 13 - see DBWR trace file
> 
> ORA-01110: data file 13: '/u1/oradata/prod/table/esis_temp_data01.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 14 - see DBWR trace file
> 
> ORA-01110: data file 14: '/u1/oradata/prod/table/esis_large_data01.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 15 - see DBWR trace file
> 
> ORA-01110: data file 15: '/u1/oradata/prod/table/esis_data01.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
> ORA-01157: cannot identify/lock data file 16 - see DBWR trace file
> 
> ORA-01110: data file 16: '/u1/oradata/prod/table/tools01.dbf'
> 
> ORA-27086: unable to lock file - already in use
> 
> Linux-x86_64 Error: 11: Resource temporarily unavailable
> 
> Additional information: 8
> 
>  
> 
> The work around that we have found is to copy all of the files off of that
> mount point and then move them back.  We even tried stopping everything
> and umounting/remounting the file system, but it still has the same
> errors.
> 
>  
> 
> Oracle and Netapp are both pointing to other as the root of the problem.
> But to me it sounds like an NFS locking issue.  Has anyone else
> experienced this, or have any suggestions on a better work around that
> having to copy the data files off and move them back?
> 

(sorry if this is a duplicate)

Im assuming you have checked to be certain no firewall is blocking any
NFS ports.  If not, do that first.  At the very least, on the client run
rpcinfo -p <server>.

Check your NFS mount parameters.  Oracle can be very picky and the error
messages are usually not very helpful.  Here is what I've found works
(YMMV):

server:/etc/exports (rw,insecure,root_squash,no_subtree_check,sync)

client:/etc/fstab  user,tcp,rsize=32768,wsize=32768,hard,intr,nfsvers=3

If the system is RAC'ed add noac to the list

The rsize and wsize can be critical under heavy I/O.

> 
> Thanks 
> 
> -Troy
> 
>  
> 


-- 
Stephen Carville <stephen at totalflood.com>
Systems Engineer
Land America
1.626.667.1450 X1326