Today, I want to share a wild ride that started as a simple video lesson, I planned to record a video on IBM BOB generating a CL program to check library sizes but quickly turned into a full-blown system emergency as I found my connection locking up, and my IBM-i system sitting at 99.9% disk usage!
If you have ever dealt with an IBM i box on the brink of crashing, this one is for you. I recorded the whole thing in real time, and after thinking about deleting it, I figured it might help someone out there facing a similar headache.
Let me break it down step by step, keeping things instructive so you can learn from my chaos. We will cover the symptoms, the diagnosis, and the fix, all focused on IBM i troubleshooting and a cool trick with AI-generated code.
How It All Started: From Lesson to Lockup
I kicked off with what was supposed to be a straightforward tutorial on some IBM i basics, maybe tweaking a few configurations or showing off a programming tip. But as I was demoing, things went sideways fast. The system started lagging, commands were hanging, and it felt like the whole machine was grinding to a halt. Classic signs of an impending crash on IBM i: response times shot up, jobs were queuing endlessly, and the console was throwing warnings left and right.
If you are familiar with IBM i, you know this platform is rock-solid, but when it hits a wall, it hits hard. In this case, the culprit was sneaky. The system was locking up because of resource exhaustion, specifically in the Integrated File System (IFS). Logs were piling up unchecked, eating disk space like there was no tomorrow. ASP (Auxiliary Storage Pool) usage was skyrocketing, and if I did not act quick, we were heading for a forced IPL (Initial Program Load) or worse.
Things to remember: Always monitor your ASP with the WRKSYSSTS command. It gives you a real-time view of CPU, disk, and memory. If you see ASP percent used creeping over 90%, alarms should be ringing.
Digging into the Problem: Real-Time Troubleshooting
As the session devolved into problem-solving mode, I jumped into detective work. First, I checked active jobs with WRKACTJOB to spot any hogs. Nothing obvious there, but subsystem activity was sluggish. Then, I poked around the IFS directories using WRKLNK, I could have used Navigator if you prefer the GUI but I wanted the green screen speed.
Sure enough, a folder bloated with old log files was the villain. These were application logs that had not been rotated or purged properly. On IBM i, the IFS is great for storing Unix-like files, but without maintenance, it can fill up storage pools fast. Common issue in environments with heavy logging from Java apps or custom programs.
Hundreds of files, gigabytes of data. The system was thrashing because it could not allocate more space for temporary objects or spooling.
The Cool Part: Using AI to Whip Up a CL Script
Here is where it got fun amidst the panic. I needed to delete those log files quickly, but manually? No way, with the system hanging. Enter BOB, my go-to AI buddy for generating code on the fly.
I described the problem:
“write a CLLE program called CLRBOBLOG which will list all files in /home/nicklitten and delete any beginning with
core.*
javacore.*
jitdump.*
Snap.*”
BOB spat out a clean CL script in seconds:
/* Program: CLRBOBLOG */
/* Purpose: List and delete specific log files from home directory */
/* Author: Nick Litten and a very naughty IBM BOB */
/* Date: 2026-02-20 */
/* This program lists all files in specified directory and deletes */
/* files matching these patterns: */
/* - core.* */
/* - javacore.* */
/* - jitdump.* */
/* - Snap.* */
/* Modification History: */
/* v.001 2026-02-20 - IBM Bob - Initial creation */
/* v.002 2026-02-21 - IBM Bob - Added HOMEDIR parameter support */
PGM PARM(&HOMEDIR)
DCL VAR(&HOMEDIR) TYPE(*CHAR) LEN(256)
DCL VAR(&CMD) TYPE(*CHAR) LEN(512)
DCL VAR(&MSGID) TYPE(*CHAR) LEN(7)
DCL VAR(&MSGDTA) TYPE(*CHAR) LEN(512)
DCL VAR(&MSGF) TYPE(*CHAR) LEN(10)
DCL VAR(&MSGFLIB) TYPE(*CHAR) LEN(10)
/* Monitor for errors */
MONMSG MSGID(CPF0000) EXEC(GOTO CMDLBL(ERROR))
/* Display header message */
SNDPGMMSG MSGID(CPF9897) MSGF(QCPFMSG) +
MSGDTA('CLRBOBLOG: Starting cleanup of log +
files in ' *CAT &HOMEDIR) TOPGMQ(*EXT) +
MSGTYPE(*INFO)
/* List all files in the directory */
SNDPGMMSG MSGID(CPF9897) MSGF(QCPFMSG) +
MSGDTA('Listing files in ' *CAT &HOMEDIR +
*TCAT ':') TOPGMQ(*EXT) MSGTYPE(*INFO)
CHGVAR VAR(&CMD) VALUE('ls -la ' *CAT &HOMEDIR)
QSH CMD(&CMD)
MONMSG MSGID(QSH0000) EXEC(DO)
SNDPGMMSG MSGID(CPF9897) MSGF(QCPFMSG) +
MSGDTA('Warning: Could not list directory +
contents') TOPGMQ(*EXT) MSGTYPE(*INFO)
ENDDO
/* Delete core.* files */
SNDPGMMSG MSGID(CPF9897) MSGF(QCPFMSG) +
MSGDTA('Deleting core.* files...') +
TOPGMQ(*EXT) MSGTYPE(*INFO)
CHGVAR VAR(&CMD) VALUE('rm -f ' *CAT &HOMEDIR *TCAT +
'/core.*')
QSH CMD(&CMD)
MONMSG MSGID(QSH0000)
/* Delete javacore.* files */
SNDPGMMSG MSGID(CPF9897) MSGF(QCPFMSG) +
MSGDTA('Deleting javacore.* files...') +
TOPGMQ(*EXT) MSGTYPE(*INFO)
CHGVAR VAR(&CMD) VALUE('rm -f ' *CAT &HOMEDIR *TCAT +
'/javacore.*')
QSH CMD(&CMD)
MONMSG MSGID(QSH0000)
/* Delete jitdump.* files */
SNDPGMMSG MSGID(CPF9897) MSGF(QCPFMSG) +
MSGDTA('Deleting jitdump.* files...') +
TOPGMQ(*EXT) MSGTYPE(*INFO)
CHGVAR VAR(&CMD) VALUE('rm -f ' *CAT &HOMEDIR *TCAT +
'/jitdump.*')
QSH CMD(&CMD)
MONMSG MSGID(QSH0000)
/* Delete Snap.* files */
SNDPGMMSG MSGID(CPF9897) MSGF(QCPFMSG) +
MSGDTA('Deleting Snap.* files...') +
TOPGMQ(*EXT) MSGTYPE(*INFO)
CHGVAR VAR(&CMD) VALUE('rm -f ' *CAT &HOMEDIR *TCAT +
'/Snap.*')
QSH CMD(&CMD)
MONMSG MSGID(QSH0000)
/* Display completion message */
SNDPGMMSG MSGID(CPF9897) MSGF(QCPFMSG) +
MSGDTA('CLRBOBLOG: Cleanup completed +
successfully') TOPGMQ(*EXT) MSGTYPE(*COMP)
GOTO CMDLBL(ENDPGM)
ERROR: /* Error handling */
RCVMSG MSGTYPE(*EXCP) MSGDTA(&MSGDTA) MSGID(&MSGID) +
MSGF(&MSGF) SNDMSGFLIB(&MSGFLIB)
SNDPGMMSG MSGID(CPF9897) MSGF(QCPFMSG) +
MSGDTA('CLRBOBLOG: Error occurred - ' *CAT +
&MSGID *CAT ' - ' *CAT &MSGDTA) +
TOPGMQ(*PRV) MSGTYPE(*COMP)
ENDPGM: ENDPGMThis uses QSH (Qshell) to run a find command that zaps old .log files. Super handy. I couldnt compile from BOB (VS CODE) because the system was shutting down its connection servers as it ran out of space – so I quickly copy/pasted into the green screen using good old SEU, compiled it with CRTBNDCL, ran it, and boom, space freed up.
Why use BOB AI? Because in a crunch, it saves time brainstorming syntax. On IBM i, blending CL with shell commands is powerful for IFS management. Just test in a safe environment first.
Wrapping It Up: System Recovery and Lessons Learned
With the logs cleared, the system breathed again. I wrapped up with a controlled PWRDWNSYS *IMMED to reboot cleanly, avoiding any data corruption risks. Post-restart, everything was golden.
Key takeaways for your IBM i toolkit:
- Set up regular housekeeping jobs for IFS logs using CL or RPG programs.
- Use tools like GO CLEANUP to automate system maintenance.
- Monitor with Performance Tools (STRPFRMON) to catch issues early.
- Experiment with AI for code gen; it is a game-changer for us programmers.
The video has all the raw action, minus the swear words I edited out for your ears. If you hit a similar lockup, give it a watch and it might help? Got questions on IBM i programming or systems? As the youngsters say Hit me up in the comments or drop a support ticket if it’s something more confidential.
Stay technical, stay troubleshooting, don’t explode and IPL like I had to 🙂