Quantitative Evaluation of Model-Driven Performance Analysis, Simulation, and Prototyping of Component-based Architectures
This page presents the results of the 2011 paper "Quantitative Evaluation of Model-Driven Performance Analysis, Simulation, and Prototyping of Component-based Architectures" by Steffen Becker, Samuel Kounev, Anne Koziolek, Heiko Koziolek, and Philipp Meier. The paper evaluates and compares different PCM analysis methods in order to obtain advantages and disadvantages of the methods and to derive guidelines for selecting the best method for a specific context. The results are based on four case studies performed under controlled conditions.
Case Studies
The evaluation uses four case studies which were introduced in earlier publications:
- Media Store: A web-based media store product line (detailed case study results | model download)
- SPECjEnterprise2010: An industry-standard benchmark designed to measure the performance of application servers conforming to the Java EE 5.0 or later specifications (Website)
- Process Control System: An industrial control system product line (detailed case study results | model download)
- Business Reporting System: A distributed business reporting system (detailed case study results | model download)
Analysis methods
The analysis methods of the PCM are used to receive performance predictions. To create a specific model used by an analysis method, model transformations are applied to a PCM instance. The PCM currently supports four analysis methods:
- SimuCom: A process-based simulation; used as reference solver.
- Layered Queueing Networks (LQN): A performance model in the class of extended queueing networks. The extension is an explicitly hierarchical structure of modeled software entities and their communication.
- Queueing Petri Nets (QPN): A combination of different extensions to conventional Petri nets.
- ProtoCom: A runnable Java/JavaEE prototype emulating real load on the underlying platform.
Comparison to SimuCom
This section presents the results for the various solvers compared to the SimuCom results. The SimuCom results serve as reference results.
Accuracy
Efficiency
Derived Guidelines
The discussion of the presented paper provides guidelines for selecting the best fitting analysis method for the practitioners' specific context:
- For general performance analysis SimuCom is recommended, as long as there is no sufficient need to reduce analysis time to justify the risk of reduced analysis accuracy or if accurate distributions of usage scenario response times are required for analysis.
- SimQPN should be used if response time distributions are needed but analysis time is important so maximum accuracy is not required.
- LQNS should be used if the prediction time should be minimized while a loss of accuracy is acceptable.
- LQNS should not be used if the PCM models use passive resources, as this can lead to high prediction errors.
- For more stability in accuracy and broader feature support, SimQPN is recommended. The analysis time can be reduced by 5 to 10% if no confidence interval for mean response times and experimental response time distributions are needed.
- If scheduling strategies other than FCFS and PS are employed, SimuCom should be used, since it allows to model scheduling strategies and these can have a significant effect on response times.
- Only SimuCom is currently able to include underlying infrastructure, like overheads for message marshalling and demarshalling with different message sizes, into the predictions as "completions".
- ProtoCom is recommended as a final validation step after simulation-based solvers generated sufficient results. ProtoCom has highter time demand needed for set up and running an usage scenario, however, it provides insights other solvers are unable to provide.