Quantitative Evaluation of Model-Driven Performance Analysis, Simulation, and Prototyping of Component-based Architectures

Aus SDQ-Wiki

This page presents the results of the 2011 paper "Quantitative Evaluation of Model-Driven Performance Analysis, Simulation, and Prototyping of Component-based Architectures" by Steffen Becker, Samuel Kounev, Anne Koziolek, Heiko Koziolek, and Philipp Meier. The paper evaluates and compares different PCM analysis methods in order to obtain advantages and disadvantages of the methods and to derive guidelines for selecting the best method for a specific context. The results are based on four case studies performed under controlled conditions.

Case Studies

Models of the case studies

The evaluation uses four case studies which were introduced in earlier publications:

Analysis methods

The analysis methods of the PCM are used to receive performance predictions. To create a specific model used by an analysis method, model transformations are applied to a PCM instance. The PCM currently supports four analysis methods:

  • SimuCom: A process-based simulation; used as reference solver.
  • Layered Queueing Networks (LQN): A performance model in the class of extended queueing networks. The extension is an explicitly hierarchical structure of modeled software entities and their communication.
  • Queueing Petri Nets (QPN): A combination of different extensions to conventional Petri nets.
  • ProtoCom: A runnable Java/JavaEE prototype emulating real load on the underlying platform.

Comparison to SimuCom

This section presents the results for the various solvers compared to the SimuCom results. The SimuCom results serve as reference results.

Accuracy

TSE-Results-Accuracy.png

Efficiency

TSE-Results-Performance.png

Derived Guidelines

The discussion of the presented paper provides guidelines for selecting the best fitting analysis method for the practitioners' specific context:

  • For general performance analysis SimuCom is recommended, as long as there is no sufficient need to reduce analysis time to justify the risk of reduced analysis accuracy or if accurate distributions of usage scenario response times are required for analysis.
  • SimQPN should be used if response time distributions are needed but analysis time is important so maximum accuracy is not required.
  • LQNS should be used if the prediction time should be minimized while a loss of accuracy is acceptable.
  • LQNS should not be used if the PCM models use passive resources, as this can lead to high prediction errors.
  • For more stability in accuracy and broader feature support, SimQPN is recommended. The analysis time can be reduced by 5 to 10% if no confidence interval for mean response times and experimental response time distributions are needed.
  • If scheduling strategies other than FCFS and PS are employed, SimuCom should be used, since it allows to model scheduling strategies and these can have a significant effect on response times.
  • Only SimuCom is currently able to include underlying infrastructure, like overheads for message marshalling and demarshalling with different message sizes, into the predictions as "completions".
  • ProtoCom is recommended as a final validation step after simulation-based solvers generated sufficient results. ProtoCom has highter time demand needed for set up and running an usage scenario, however, it provides insights other solvers are unable to provide.