Thursday, November 6, 2014

Exadata, DBFS Filesystem and ORA-64007: invalid store specified

The Activity was to resize /dbfsmnt mount point from 2 TB to 1 TB due to space crunch in DATA Diskgroup started showing in -ve (Negative) Value. Our DBFS Database had 2TB Datafile/Tablespace exist in DATA Diskgroup.

NAME    STATE Usable Total DG Space (GB) Usable Free DG Space (GB) OFFLINE_DISKS
-------------------- ----------- -------------------------- ------------------------- -------------
DATA      MOUNTED      13248   -52.699219  0
DBFS      MOUNTED   135.1875   134.539063  0
RECO      MOUNTED 3312.65625    1744.9082  0

Steps I carried out to remove existing DBFS File System,

Stop the dbfs_mount service in clusterware using the oracle account.

[inmumdcdbadm01.tajhotels.com -> oracle:/home/oracle] :crsctl stop resource dbfs_mount
CRS-2673: Attempting to stop 'dbfs_mount' on 'inmumdcdbadm02'
CRS-2673: Attempting to stop 'dbfs_mount' on 'inmumdcdbadm01'
CRS-2677: Stop of 'dbfs_mount' on 'inmumdcdbadm02' succeeded
CRS-2677: Stop of 'dbfs_mount' on 'inmumdcdbadm01' succeeded

[inmumdcdbadm01.tajhotels.com -> oracle:/home/oracle] :crsctl stat resource dbfs_mount -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
dbfs_mount
               OFFLINE OFFLINE      inmumdcdbadm01
               OFFLINE OFFLINE      inmumdcdbadm02

Drop DBFS File System

[oracle@inmumdcdbadm01 oradbfs]$ cat /home/oracle/dbs_group
inmumdcdbadm01
inmumdcdbadm02

[inmumdcdbadm01.tajhotels.com -> root:/root] :dcli -g /home/oracle/dbs_group -l root rmdir /dbfsmnt

[inmumdcdbadm01.tajhotels.com -> oracle://home/oracle] :. ./DBFS1.env
[inmumdcdbadm01.tajhotels.com -> oracle:/u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/admin] :sqlplus dbfs_user/******

SQL*Plus: Release 11.2.0.3.0 Production on Fri Oct 21 18:16:32 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> @dbfs_drop_filesystem oradbfs
No errors.
--------
UNMOUNT STORE:
begin dbms_dbfs_content.unmountStore(store_name=>'FS_ORADBFS',
store_mount=>'oradbfs'); end;
--------
UNREGISTER STORE:
begin dbms_dbfs_content.unregisterStore(store_name=> 'FS_ORADBFS'); end;
--------
DROP STORE:
begin dbms_dbfs_sfs.dropFilesystem(store_name => 'FS_ORADBFS'); end;
No errors.

Drop DBFS User

[inmumdcdbadm01.tajhotels.com -> oracle:/u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/admin] :sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Fri Oct 21 18:22:28 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> select name from v$database;

NAME
---------
DBFS

SQL> show user
USER is "SYS"

SQL> drop user DBFS_USER cascade;

User dropped.

SQL>  alter database datafile '+DATA_DC/dbfs/datafile/dbfsts.274.828974767' resize 1024G;

Database altered.

Recreate DBFS User

[inmumdcdbadm01.tajhotels.com -> root:/root] :dcli -g /home/oracle/dbs_group -l root mkdir /dbfsmnt

SQL> create user dbfs_user identified by ****** default tablespace dbfsts quota unlimited on dbfsts;

User created.

SQL> grant create session, create table, create view, create procedure, dbfs_role to dbfs_user;

Grant succeeded.

Recreate DBFS Filesystem

[oracle@inmumdcdbadm01 ~]$ . ./DBFS1.env
[oracle@inmumdcdbadm01 ~]$ cd $ORACLE_HOME/rdbms/admin
[oracle@inmumdcdbadm01 admin]$ sqlplus dbfs_user/*****

SQL*Plus: Release 11.2.0.3.0 Production on Wed Oct 21 19:33:03 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> @dbfs_create_filesystem.sql dbfsts oradbfs

No errors.
--------
CREATE STORE:
begin dbms_dbfs_sfs.createFilesystem(store_name => 'FS_ORADBFS', tbl_name =>
'T_ORADBFS', tbl_tbs => 'dbfsts', lob_tbs => 'dbfsts', do_partition => false,
partition_key => 1, do_compress => false, compression => '', do_dedup => false,
do_encrypt => false); end;
ERROR: -64007 msg: ORA-64007: invalid store specified
ORA-06512: at
"SYS.DBMS_DBFS_SFS_ADMIN", line 1378
ORA-00001: unique constraint
(SYS.SYS_C003273) violated
declare
*
ERROR at line 1:
ORA-64007: invalid store specified
ORA-06512: at "SYS.DBMS_DBFS_SFS_ADMIN", line 1378
ORA-00001: unique constraint (SYS.SYS_C003273) violated
ORA-06512: at line 63

No errors.

Solution for above issue

[oracle@inmumdcdbadm01 admin]$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.3.0 Production on Wed Nov 5 20:31:22 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> select owner, table_name from dba_tables where table_name like '%T_ORADBFS%';

no rows selected

SQL> select tabid from sys.dbfs_sfs$_tab where table_name = 'T_ORADBFS';

     TABID
----------
         1

SQL> delete from sys.dbfs_sfs$_vol where tabid = 1;

1 row deleted.

SQL> delete from sys.dbfs_sfs$_tab where tabid = 1;

1 row deleted.

SQL> commit;

Commit complete.

SQL> conn dbfs_user
Enter password:
Connected.
SQL> !pwd
/u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/admin

SQL> !ls -l dbfs_create_filesystem.sql
-rw-r--r-- 1 oracle oinstall 974 Jun  1  2010 dbfs_create_filesystem.sql

SQL> @dbfs_create_filesystem.sql dbfsts oradbfs
No errors.
--------
CREATE STORE:
begin dbms_dbfs_sfs.createFilesystem(store_name => 'FS_ORADBFS', tbl_name =>
'T_ORADBFS', tbl_tbs => 'dbfsts', lob_tbs => 'dbfsts', do_partition => false,
partition_key => 1, do_compress => false, compression => '', do_dedup => false,
do_encrypt => false); end;
--------
REGISTER STORE:
begin dbms_dbfs_content.registerStore(store_name=> 'FS_ORADBFS', provider_name
=> 'sample1', provider_package => 'dbms_dbfs_sfs'); end;
--------
MOUNT STORE:
begin dbms_dbfs_content.mountStore(store_name=>'FS_ORADBFS',
store_mount=>'oradbfs'); end;
--------
CHMOD STORE:
declare m integer; begin m := dbms_fuse.fs_chmod('/oradbfs', 16895); end;
No errors.
SQL>

[oracle@inmumdcdbadm01 admin]$ cd
[oracle@inmumdcdbadm01 ~]$ . ASM.env

[oracle@inmumdcdbadm01 ~]$ sh ./add-dbfs-resource.sh
CRS-2539: A resource with the name 'dbfs_mount' is already registered
CRS-4000: Command Add failed, or completed with errors.

[oracle@inmumdcdbadm01 ~]$ crsctl stat resource dbfs_mount -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
dbfs_mount
               OFFLINE OFFLINE      inmumdcdbadm01
               OFFLINE OFFLINE      inmumdcdbadm02

[inmumdcdbadm01.tajhotels.com -> oracle:/home/oracle] :crsctl start resource dbfs_mount
CRS-2672: Attempting to start 'dbfs_mount' on 'inmumdcdbadm02'
CRS-2672: Attempting to start 'dbfs_mount' on 'inmumdcdbadm01'
CRS-2676: Start of 'dbfs_mount' on 'inmumdcdbadm02' succeeded
CRS-2676: Start of 'dbfs_mount' on 'inmumdcdbadm01' succeeded

[inmumdcdbadm01.tajhotels.com -> oracle:/dbfsmnt] :df -k /dbfsmnt
Filesystem           1K-blocks      Used Available Use% Mounted on
dbfs-dbfs_user@:/    524285952       120 524285832   1% /dbfsmnt

[inmumdcdbadm01.tajhotels.com -> oracle:/dbfsmnt] :ssh oracle@inmumdcdbadm02 df -k /dbfsmnt
Filesystem           1K-blocks      Used Available Use% Mounted on
dbfs-dbfs_user@:/    524285952       120 524285832   1% /dbfsmnt

Sometime it is required to enable trace to diagnose/troubleshoot issue related to DBFS Mount Point / File System w.r.t Cluster Services, hence do the following,

vi $GI_HOME/crs/script/mount-dbfs.sh
MOUNT_OPTIONS=trace_level=1,trace_file=/tmp/dbfs_client_trace.$$.log,trace_size=100

Now start the resource one more time to get the log file generated.

[oracle@inmumdcdbadm01 ~]$ dbfs_client dbfs_user@ -o allow_other,direct_io,trace_level=1,trace_file=/tmp/dbfs_client_trace.$$.log /dbfsmnt
Password:
Fail to connect to database server.

[oracle@inmumdcdbadm01 ~]$ tail /tmp/dbfs_client_trace.11083.log.0

[66529940 11/05/14 21:01:44.529499 ./LcdfThreadPool.h:276     ] DEBUG: Retrying after some time .110. 1415201504 542132553
[6fd3c940 11/05/14 21:01:48.634994 LcdfDBPool.cpp:189         ] ERROR: Failed to create session pool ret:-1
[6fd3c940 11/05/14 21:01:48.635069 LcdfDBPool.cpp:399         ] ERROR: ERROR 15000 - ORA-15000: command disallowed by current instance type

[6fd3c940 11/05/14 21:01:48.635090 LcdfDBPool.cpp:251         ] DEBUG: Clean up OCI session pool...
[6fd3c940 11/05/14 21:01:48.635315 LcdfDBPool.cpp:399         ] ERROR: ERROR 24416 - ORA-24416: Invalid session Poolname was specified.

[6fd3c940 11/05/14 21:01:48.635671 LcdfDBPool.cpp:444         ] CRIT : Fail to set up database connection.

----------------------------------------------------------------------------------------

tail /tmp/dbfs_client_trace.100641.log.0

 [43b6c940 03/12/14 11:15:01.577723 LcdfDBPool.cpp:189         ] ERROR: Failed to create session pool ret:-1
 [43b6c940 03/12/14 11:15:01.577753 LcdfDBPool.cpp:399         ] ERROR: ERROR 28001 - ORA-28001: the password has expired

[43b6c940 03/12/14 11:15:01.577766 LcdfDBPool.cpp:251         ] DEBUG: Clean up OCI session pool...
 [43b6c940 03/12/14 11:15:01.577805 LcdfDBPool.cpp:399         ] ERROR: ERROR 24416 - ORA-24416: Invalid session Poolname was specified.

[43b6c940 03/12/14 11:15:01.577844 LcdfDBPool.cpp:444         ] CRIT : Fail to set up database connection.

Useful Links which I Referred,

http://fritshoogland.wordpress.com/2010/07/17/dbfs-and-ora-64007-invalid-store-specified/

http://sve.to/2014/03/13/troubleshooting-oracle-dbfs-mount-issues/

By doing this Activity, I could able to accumulate approx 960 GB Free Space in DATA Diskgroup. 

NAME    STATE Usable Total DG Space (GB) Usable Free DG Space (GB) OFFLINE_DISKS
-------------------- ----------- -------------------------- ------------------------- -------------
DATA      MOUNTED      13248   959.773438  0
DBFS      MOUNTED   135.1875   134.539063  0
RECO      MOUNTED 3312.65625   1708.78516  0


I hope this might be helpful to someone. :)

Thanks,
Manish

No comments:

Post a Comment