Internships/ProjectIdeas/QEMUPerformance: Difference between revisions

Latest revision as of 10:42, 5 February 2020

TCG Continuous Benchmarking

Summary: The nature of this project lies more in exploration, analysis and presentation than in coding. The performance of a software product will be examined to the greatest details. The software under examination will be QEMU emulator - across its modes, across its components, and across time.

QEMU may operate in so called user mode (an executable built for one processor (in QEMU parlance, target) is, by means of QEMU emulation, executed on the system with another processor (again, in QEMU slang, host)) and in system mode (the whole system of one kind (target) is emulated on the system of another kind (host)). These two modes will be examined separately:

TASK

PART I: (user mode)

select around a dozen test programs (resembling components of SPEC benchmark, but all must be open source, and preferably having license compatible with QEMU); those test programs should be distributed like this: 4-5 FPU CPU-intensive, 4-5 non-FPU CPU intensive, 1-2 I/O intensive;
measure execution time and other performance data of all selected test program across all targets on Intel and possibly other hosts, for the latest QEMU version:
- try to improve performance if there is an obvious bottleneck;
- develop tests that will be protection against performance regressions in future;
- provide automated nightly tests for letting know QEMU developers if something changed performance-wise.
measure execution time of all selected test programs for selected targets for all QEMU versions in last 5 years (there are appr. 15 such versions):
- confirm performance improvements and/or detect performance degradation.
summarize all results in a comprehensive form, using also graphics/data visualization.

PART II: (system mode)

measure execution time and other performance data for boot/shutdown cycle for selected machines for the latest QEMU version:
- try to improve performance if there is an obvious bottleneck.
summarize all results in a comprehensive form.

DELIVERABLES

1) Each target maintainer for target will be given a list of top 25 functions in terms of spent host CPU time for each benchmark described in the previous section. Additional information and observations will be also provided, if the judgment is they are useful and relevant.

2) Each machine maintainer machine (that has successful boot/shutdown cycle) will be given a list of top 25 functions in terms of spent host time during boot/shutdown cycle. Additional information and observations may also be provided.

3) The community will be given all devised performance measurement methods in the form of easily reproducible step-by-step setup and execution procedures.

Deliverables should be gradually distributed over wider time interval of around two months.

Links:

Details:

Skill level: intermediate
Languages:
- C (for code analysis, performance improvements)
- Python (for automatization)
- potentially JavaScript (d3.js or similar library; for data visualization)
Mentor: Aleksandar Markovic (aleksandar.markovic@rt-rk.com)
Suggested by: Aleksandar Markovic

@@ Line 1: / Line 1: @@
+=== TCG Continuous Benchmarking ===
-'''Summary:''' The nature of this project is the exploration and analysis of performance of a software tool. The tool in this case is QEMU, and its two modes of operation: user mode and system mode
+'''Summary:''' The nature of this project lies more in exploration, analysis and presentation than in coding. The performance of a software product will be examined to the greatest details. The software under examination will be QEMU emulator - across its modes, across its components, and across time.
+QEMU may operate in so called user mode (an executable built for one processor (in QEMU parlance, ''target'') is, by means of QEMU emulation, executed on the system with another processor (again, in QEMU slang, ''host'')) and in system mode (the whole system of one kind (target) is emulated on the system of another kind (host)). These two modes will be examined separately:
+'''TASK'''
 PART I: (user mode)
-* select around a dozen test programs (resembling components of SPEC benchmark, but must be open source, and preferably license compatible with QEMU); test programs should be distributed like this: 4-5 FPU CPU-intensive, 4-5 non-FPU CPU intensive, 1-2 I/O intensive;
+* select around a dozen test programs (resembling components of SPEC benchmark, but all must be open source, and preferably having license compatible with QEMU); those test programs should be distributed like this: 4-5 FPU CPU-intensive, 4-5 non-FPU CPU intensive, 1-2 I/O intensive;
-* measure execution time and other performance data in user mode across all platforms for latest QEMU version:
+* measure execution time and other performance data of all selected test program across all targets on Intel and possibly other hosts, for the latest QEMU version:
-       - try to improve performance if there is an obvious bottleneck;
+** try to improve performance if there is an obvious bottleneck;
-       - develop tests that will be protection against performance regressions in future.
+** develop tests that will be protection against performance regressions in future;
-* measure execution time in user-mode for selected platforms for all QEMU versions in last 5 years:
+** provide automated nightly tests for letting know QEMU developers if something changed performance-wise.
-       - confirm performance improvements and/or detect performance degradation.
+* measure execution time of all selected test programs for selected targets for all QEMU versions in last 5 years (there are appr. 15 such versions):
+** confirm performance improvements and/or detect performance degradation.
 * summarize all results in a comprehensive form, using also graphics/data visualization.
 PART II: (system mode)
-* measure execution time and other performance data for boot/shutdown cycle for selected machines for ToT:
+* measure execution time and other performance data for boot/shutdown cycle for selected machines for the latest QEMU version:
-       - try to improve performance if there is an obvious bottleneck;
+** try to improve performance if there is an obvious bottleneck.
-       - develop tests that will be protection against performance regressions in future.
 * summarize all results in a comprehensive form.
-DELIVERABLES
-) Each maintainer for target will be given a list of top 25 functions in terms of spent host time for each benchmark described in the previous section. Additional information and observations will be also provided, if the judgment is they are useful and relevant.
+'''DELIVERABLES'''
-) Each maintainer for machine (that has successful boot/shutdown cycle) will be given a list of top 25 functions in terms of spent host time during boot/shutdown cycle. Additional information and observations will be also provided, if the judgment is they are useful and relevant.
+) Each target maintainer for target will be given a list of top 25 functions in terms of spent host CPU time for each benchmark described in the previous section. Additional information and observations will be also provided, if the judgment is they are useful and relevant.
+) Each machine maintainer machine (that has successful boot/shutdown cycle) will be given a list of top 25 functions in terms of spent host time during boot/shutdown cycle. Additional information and observations may also be provided.
 ) The community will be given all devised performance measurement methods in the form of easily reproducible step-by-step setup and execution procedures.
-Deliverable should be gradually distributed over wider time interval of around two months.
+Deliverables should be gradually distributed over wider time interval of around two months.
 '''Links:'''
@@ Line 34: / Line 43: @@
 * [https://en.wikipedia.org/wiki/List_of_performance_analysis_tools]
 * [https://www.tableau.com/learn/articles/data-visualization]
 '''Details:'''