Tuesday, August 12, 2014

SAP BPC - Catch-me-if-you-can problems during planning cycle


Reports used to run normally and then it would start crawling such as taking 5 to 10 mins. to complete. When one starts to check things out from IT, the problem usually goes away and reports were back to normal runtime. This problem used to aggravate towards the end of the planning cycle, and we concluded that it was due to the load as everyone jammed the system. This is the angle we pursued for some time, but could not nail it.

After a few days, we started learning from IIS logs, that this problem was evident even when one or two users were on the system. This invalidated the theory that the load was the sole reason for the hiccups.

After some more digging, we were able to tie the occurrence of the problem to the "stats" health light SAP BW was displaying for a cube under Manage Action/Performance tab.
With that, we started running "rsvc" transactions at periodic intervals during busy days.
Earlier runs of rsvc were reporting green whenever it was run, so did not think this was the likely cause.

Once we started running it more often, we netted the problem - the "F-Fact" (temporary or new record) table of a cube was having invalid stats when an input template was submitted.
The "E-Fact" (more permanent or old record) table of the same cube was having up-to-date stats, and it was not impacted at all.

Trapping some of the report queries from Oracle Grid Control showed that records from both "E and F" Fact tables were joined with master data dimension tables.
It was guessed that Oracle optimizer could not do well when it ran into a huge table(E-Fact - 20+ M records) with good stats and small tables (F-Fact - 4K records) with invalid stats.

So approached the DBAs to press a re-compute stats job on the affected F-Fact table every 15 mins and the problem is pretty much gone since then.
       execute dbms_stats.gather_table_stats(..., ...);

If it were to happen again, decided to increase the frequency to every 10 mins. or even 5 mins.

No comments: