Thursday, February 21, 2013

Help! APEX is blowing up our memory!

That's the message I received from a DBA friend who ran into this problem at her customer site. What really happened was that a couple of times a week one of the APEX processes was killed by the operating system due to reaching the limit of 16Gb memory. 16Gb!
And this resulted in an ORA-04030 message in the Oracle Database.
So I went to that client site and we dove into the database log files, the APEX logging, the access log files and the error log files of the web server. Every time we noticed calls of WWW_FLOW_FILE_MGR.GET_FILE that seemed to be the problem. But the application wasn't huge, just 25 pages. Nothing complex, all pretty straight forward. So we started to look for the origin of those calls.
Just like in a lot of applications, this application also had a few images on the top of the screen (like Home, Logout, etc.). Just four or five on every page. However, these images were stored inside the APEX Repository, using the #WORKSPACE_IMAGES# substitution string. And that results in a call to WWW_FLOW_FILE_MGR.GET_FILE for every image. So that's five requests for every page, while it could - and should - be only one. And four executions of that procedure every time, while it could - and should - be zero. So I advised to move these images to the web server, so they can be cached and wouldn't result in hitting the database anymore.
But when this would be the cause of the problem, it made sense it would occur on peak times. But the problem didn't occur on peak times. The last one was that same morning around 4:30AM.
So we dove further and then we discovered that quite a few of the select statements in the application used #WORKSPACE_IMAGES# as well! So the procedure would be executed for every row returned by the select statement. So in some particular cases - just happening three times a week or so - the query would return 10,000 of rows. Or even millions. So the procedure would be executed that often. And the best thing is: the image that the select was trying to retrieve ... didn't even exists! So millions of calls to the database to retrieve ... nothing!
So changing the select statements and moving all images to the web server will reduce the number of calls to that procedure from millions (or 100 of millions) times a week to ... zero (?).
And the chances are the memory problem is gone to.

So when you run into these kind of problems were APEX seems to consume a lot of memory - or CPU for that matter - take a very good look to your own code first. The chances are that the resource consumption is not because APEX is misbehaving, but your code isn't optimal....
Post a Comment