dbnodeupdate Warning: Total amount of free space in the volume group is less than 2GB. Unable to make a snapshot

Asset ID:	1-72-1644872.1
Update Date:	2014-04-09
Keywords:

Solution Type Problem Resolution Sure

Solution 1644872.1 : dbnodeupdate Warning: Total amount of free space in the volume group is less than 2GB. Unable to make a snapshot

Applies to:

Exadata Database Machine V2 - Version All Versions to All Versions [Release All Releases]
Linux x86-64

Symptoms

When running dbnodeupdate.sh the following warning message is presented:

Warning: Total amount of free space in the volume group is less than 2GB. Unable to make a snapshot
Warning: Backup functionality will not be provided by dbnodeupdate.sh and need to be made separately
Continue ? [Y/n]

Example:

./dbnodeupdate.sh -u -l /u01/patches/YUM/p17809253_112330_Linux-x86-64.zip -v
##########################################################################################################################
.

.
Continue ? [Y/n]
y
(*) 2014-04-06 20:20:04: Unzipping helpers (/u01/patches/YUM/dbupdate-helpers.zip) to /opt/oracle.SupportTools/dbnodeupdate_helpers
(*) 2014-04-06 20:20:04: Initializing logfile /var/log/cellos/dbnodeupdate.log
(*) 2014-04-06 20:20:05: Collecting system configuration details. This may take a while...
(*) 2014-04-06 20:20:17: Validating system details for known issues and best practices. This may take a while...
(*) 2014-04-06 20:20:17: Checking free space in /u01
(*) 2014-04-06 20:20:17: Unzipping /u01/patches/YUM/p17809253_112330_Linux-x86-64.zip to /u01/app/oracle/stage.060414202002, this may take a while
(*) 2014-04-06 20:20:34: Original /etc/yum.conf moved to /etc/yum.conf.060414202002, generating new yum.conf
(*) 2014-04-06 20:20:34: Generating Exadata repository file /etc/yum.repos.d/Exadata-computenode.repo
Warning: Total amount of free space in the volume group is less than 2GB. Unable to make a snapshot
Warning: Backup functionality will not be provided by dbnodeupdate.sh and need to be made separately
Continue ? [Y/n]

Changes

Applying a new version of the Exadata software (upgrade).

Cause

Not enough free space (at least 2GB) in the volume group for a backup snapshot to be taken.

This is because the logical volume associated with the /u01 file system must have been manually edited to increase useable space thus leaving very little free space at the volume group level.

This could be seen on one or many nodes.

This space problem can be confirmed by running the vgs command

example:

[root@ExaDB01]# vgs

VG #PV #LV #SN Attr VSize VFree

VGExaDb 1 4 0 wz--n- 556.43G 436.00M <=== Volume Group VGExaDb only has 436MB of free space.

View all LV's and sizes

[root@ExaDB01 ~]# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
LVDbOra1 VGExaDb -wi-ao 472.00G
LVDbSwap1 VGExaDb -wi-ao 24.00G
LVDbSys1 VGExaDb -wi-ao 30.00G
LVDbSys2 VGExaDb -wi-a- 30.00G

In this example, the problem is not seen on the other node in the Cluster

Output from good node ExaDB02

[root@ExaDB02]# vgs

VG #PV #LV #SN Attr VSize VFree

VGExaDb 1 4 0 wz--n- 556.43G 12.43G <==Good node 12G of Free space

[root@ExaDB02 ~]# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
LVDbOra1 VGExaDb -wi-ao 460.00G <==Good node shows LVDbOra1 (/u01 partition) is configured with 12G less than the bad node (DB01) seen above
LVDbSwap1 VGExaDb -wi-ao 24.00G
LVDbSys1 VGExaDb -wi-ao 30.00G
LVDbSys2 VGExaDb -wi-a- 30.00G

Solution

Need to reduce the size of the Logical Volume associated with the /u01 partition by 12GB so it matches with the good node showing 12GB of free space.

1) Find the LV name of assigned to the /u01 partition

[root@ExaDB01 ~]# df -kvh
Filesystem            Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbSys1
                       30G   19G 9.9G 65% /
/dev/sda1             502M 111M 366M 24% /boot
tmpfs                  81G     0   81G   0% /dev/shm
/dev/mapper/VGExaDb-LVDbOra1                     <==LVDbOra1 is the LV name
                      472G 113G 359G 31% /u01

2) Backup all data on /u01 in case of data loss during reduce procedure.

3) Stop all applications utilizing /u01. In most cases this means stoping Databases and CRS. And un-mount the /u01 file system

example: umount /u01

4) Run File System check

example: [root@ExaDB01 ~]# e2fsck -f /dev/mapper/VGExaDb-LVDbOra1
e2fsck 1.39 (29-May-2006)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

5) Reduce the size of the file system by 12G

example: resize2fs /dev/mapper/VGExaDb-LVDbOra1 460G
resize2fs 1.39 (29-May-2006)
Resizing the filesystem on /dev/mapper/VGExaDb-LVDbOra1 to 120586240 (4k) blocks.

The filesystem on /dev/mapper/VGExaDb-LVDbOra1 is now 120586240 blocks long.

6) Now reduce the size of the LV by 12G

example: [root@ExaDB01 ~]# lvm
lvm> lvreduce -L 460G /dev/mapper/VGExaDb-LVDbOra1
WARNING: Reducing active logical volume to 460.00 GB
THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce LVDbOra1? [y/n]: y
Reducing logical volume LVDbOra1 to 460.00 GB
Logical volume LVDbOra1 successfully resized

7) mount the newly resized /u01 partition (LVDbOra1)

example: [root@ExaDB01 ~]# mount -t ext3 /dev/mapper/VGExaDb-LVDbOra1 /u01

8) rerun vgs command to confirm free space

example: [root@ExaDB01]# vgs

VG #PV #LV #SN Attr VSize VFree

VGExaDb 1 4 0 wz--n- 556.43G 12.43G <=== Volume Group now shows 12G of free space

9) With free space issue now resolved re-run upgrade actions (dbnodeupdate.sh)

References

<NOTE:1553103.1> - Exadata Database Server Patching using the DB Node Update Utility

Attachments

This solution has no attachment