Veeam RMAN strange errors: always check for disk space first :-D
Veeam RMAN strange errors: always check for disk space first :-D
Today I had to investigate reported errors on a *RMAN Plugin backup *failing with errors like this
Backing up data
RMAN error: RMAN-03002: failure of backup plus archivelog command at 04/23/2025 01:15:35
ORA-19506: failed to create sequential file,
name="67f74238-2cef-4d85-bd0a-1166a0598955/RMAN_11409181_WP_20250423_gv3nj518_416287_1_1.vab",
parms="" ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: non RMAN, but media manager or vendor specific failure,
error text: TCP stream was closed --tr:Failed to start source agent.
--tr:Failed to start shared source agent. --tr:Failed to get agents session. --tr
“oh my god, I’m going to collect a tons of logs, ask help from the Oracle DBa, open a case…”
But yesterday I had a very nice evening and this morning my mind was fresh… so I issued this command on the Oracle server
[root@ora11 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 40G 0 40G 0% /dev/shm
tmpfs 16G 1.7G 14G 11% /run
/dev/mapper/ol-root 66G 20G 47G 30% /
/dev/mapper/ol-home 32G 12G 21G 37% /home
/dev/sdb2 960M 446M 515M 47% /boot
/dev/sdb1 599M 6.3M 593M 2% /boot/efi
/dev/mapper/VolGroup02-LogVol03 15G 15G 0 100% /var
/dev/mapper/VolGroup00-LogVol01 98G 44G 50G 47% /app
/dev/mapper/VolGroupOradata-Oradata 2.0T 1.4T 546G 72% /oradata
tmpfs 7.9G 0 7.9G 0% /run/user/0
tmpfs 7.9G 0 7.9G 0% /run/user/1343843400
hey, /var is 100% full!!!
this was preventing the RMAN plugin to work properly. Freeing up space was the solution ;-)
P.S.: 15 GBytes for the /var is absolutely not enough. I resized the volume to 50 GBytes and edited the /etc/logrotate.conf to reduce the retention from 4 to 3 weeks (messages rotated logs were taking almost all of the 15 GBytes)
[root@ora11 log]# cat /etc/logrotate.conf
# see "man logrotate" for details
# global options do not affect preceding include directives
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 3
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
dateext
# uncomment this if you want your log files compressed
#compress
# packages drop log rotation information into this directory
include /etc/logrotate.d
# system-specific logs may be also be configured here.