Problem
ProNews is one of the ProSiebenSat.1 products. It represents a newsroom system for news editorial boards where journalists create TV-programs. ProNews consists of the following modules:
- Newsticker, collects and supplies reports of news agencies;
- Rundown, represents overview of TV-programs for a single day;
- Beitragspool, a text editor to compose news articles;
- Storyboard, searches and previews video materials.
The following problem has been met during operation of ProNews: when hundreds of clients worked intensively with the system during the day, the amount of memory required for server functioning grew constantly. Till midnight the amount of allocated memory became so huge that it caused sufficient performance degradation. Approximately at 2 A.M. every day (when system load was minimal) technicians restarted ProNews. It was obvious that the system suffered from memory leaks, but hunting them was not easy due to the great size of the application.
Solution
Searching memory leaks in such an application is a serious challenge. Due to the size of the system it was almost impossible to start it under profiler; moreover the amount of profile data retrieved seemed to be non-analyzable. We have concentrated on those parts of the system that participated in all clients' requests and had maximal load. Source code inspection showed only high professional level of the system developers. We've worked out a hypothesis that only extreme load resulted to memory leaks. To examine it we've isolated events dispatching subsystem and developed test environment emulating activity of hundreds of clients. Experiments showed that memory leaks appeared after several hours of extreme load of events dispatching subsystem. Further investigation with use of profiler detected an error in third-party middleware library. This error was caused by incorrect optimization under ultimate load. After fixing this fault, problems with growing memory consumption disappeared.
