| |
Language Performance Test
Robert A. Wlodarczyk
Microsoft Student Consultant, SUNY Stony Brook
Overview
During the course of the Fall 2001, people throughout the computing industry have been learning more and more about Microsoft’s .NET Platform as a whole. We’ve seen a number of articles talking about how .NET will revolutionize the way the entire industry will develop applications over the next few years. Applications will become more and more ubiquitous, with users accessing their data from virtually anyplace in the globe off any device – desktop PC, tablet pc, PDA, or cell phone. .NET will be among the primary driving forces behind this.
Purpose
With the introduction of the .NET Framework, a few new languages, as well as new versions of existing languages are being released. It is important to take a look at how these languages stack up against languages and versions that currently exist and that many people are familiar with. In addition to see how all these languages compare, with the advent of the .NET Framework, applications are compiled to run on top of the Common Language Runtime (CLR). Here, applications run in a managed environment providing certain basic operations for your applications – type safety, code verifiability, memory management, thread management, and security. One important distinction between the CLR and other similar types of architectures, such as the Java Virtual Machine (JVM) or the Visual Basic Virtual Machine (VBVM), is that your code is never interpreted. This means that your code is being run at native machine speed. Such a performance test will be able to show that indeed this is the case. Finally, such a test will provide results showing the similarities and differences in speed between .NET languages and those that many are already familiar with.
How the test was performed
In this section, the details of the test will be discussed. This includes what languages and versions were used for the testing, as well as a description of the test machine. Finally, the way the tests were executed will be explained.
Languages Covered
| Language |
Version |
| Sun Java |
1.3.1_01 |
| Microsoft C# |
1.0 (Build 9466, RTM) |
| Microsoft Visual Basic |
6.0 (SP5) |
| Microsoft Visual Basic .NET |
7.0 (Build 9466, RTM) |
| Microsoft Visual C++ |
6.0 (SP5) |
| Microsoft Visual C++ .NET |
7.0 (Build 9466, RTM) |
Table 1: List of all languages and versions used.
Test Machine
All tests were performed on a mid-range desktop consisting of the following setup:
- ASUS P2B-F Revision 1011 Motherboard
- Intel i440BX Chipset
- Pentium III 600
- 256Mb RAM, PC100, CAS #3
Testing was performed using the following software setup:
- Windows 2000 Professional, Service Pack 2
- Internet Explorer 6.0, all updates as of 12/21/2001
- Visual Studio 6.0, Service Pack 5
- Visual Studio .NET, Build 9466, RTM
Testing Code
For each of the tests, the essential basics of the performance are identical. Keeping such a performance test simple provides the end user with an easy way of interpreting the results. The core of each of the area of testing encompassed the use of the same code.
double tmrStart, tmrElapsed;
int i, j;
tmrStart = GetCurrentTime();
for(i = 0; i < 10000; i++)
{
for(j = 0; j < 10000; j++)
{
{do whatever test here}
}
}
tmrElapsed = GetCurrentTime() - tmrStart;
Print(tmrElapsed);
|
Figure 1: Code framework used for all of the tests.
Since, in general, any single operation takes a considerably small amount of time, in order to be able to see a performance of a certain operation it must be repeated many times. In the case of these tests, a repetition of 100,000,000 was chosen. With such a high repetition rate, operations that would normally be nearly instantaneous take a considerable amount of time to complete.
When designing such tests, it is important to remember that the code that is adding in the {do whatever test here} section in Figure 1 must be written in such a way that the compiler cannot modify its operation. For example, if that code were x = 5, then most well optimizing compilers will remove the two for loops and just say x = 5 once. So, this code must be something that the compiler won’t have anyway to reduce to one operation.
Areas Covered
For purposes of this test, we looked at a number of different basic areas. Each area looks at potentially different implementations within the compilers.
| Type of Test |
Class of Test |
| Integer Addition |
Integer Arithmetic |
| Integer Multiplication |
Integer Arithmetic |
| Double Addition |
Double Arithmetic |
| Double Multiplication |
Double Arithmetic |
| Object Instantiation |
Object Creation |
| Object Property Get |
Object Use |
| Object Property Set |
Object Use |
| Object Method Call |
Object Use |
Table 2: List of types of test and their classifications.
Execution of Tests
All tests were run using an execution script that was written using VBScript and the Windows Scripting Host. All tests were repeated fifty times in order to reduce the variability of execution times. This provided us with variability of less than 2.5%, but on average less than 1.7%. Thus, our results have a very high confidence level. In Figure 2, you can see the general idea of how the VBScript was written to run all the performance tests.
for each language
for each test type
for 1 to N
{perform the test}
{wait M seconds}
next
{print the results from all the runs}
next
{wait Q seconds}
next
|
Figure 2: General schema of how the tests were executed.
For our particular instance, we chose M to be 10 seconds, and Q to be 150 seconds (2.5 minutes). The M time interval allows the system to “rest” for a brief period of time before running the test again. The Q time interval allows the system to “recover” from a particular language’s full run of tests. This may include memory cleanup or other such things. Essentially, this is provided to stabilize the system enough so that the next language is running under similar conditions as the first language did.
Results of the Test
All of these tests running on the machine, all as described above, took approximately 30 hours to run. Each type of test was later analyzed and compared to each other. In brief, we found that C# code is on average 3.30 times faster than native Java code. An interesting benchmark we found was that C# was on average 3.11 times faster than Visual C++ 6.0. Also, Visual Basic .NET code ran 46.45 times faster, on average, than Visual Basic 6.0 code.
On a category basis, we find that C# is 2.68 times faster in integer arithmetic than Java, 1.46 times faster in double arithmetic than Java, 1.93 times faster in object instantiation than Java, and 5.38 times faster in object use than Java.
A listing of all the average runtimes for all the languages and types of test are in Table 3. Also provided is a table listing the variability for each of the tests performed. These results can be seen in Table 4. For graphs of the data, take a look at Graph 1, Graph 2, and Graph 3.
|
Integer Addition |
Double Addition |
Integer Multiplication |
Double Multiplication |
Object Instantiation |
Object Property Get |
Object Property Set |
Object Method Call |
| C# |
505 |
1515 |
506 |
1517 |
5349 |
335 |
334 |
334 |
| VB.NET |
342 |
6078 |
343 |
6074 |
5183 |
342 |
342 |
677 |
| VC++.NET |
0 |
1653 |
0 |
1655 |
85111 |
0 |
1093 |
3070 |
| VB 6.0 |
670 |
1892 |
1346 |
2529 |
1212860 |
18705 |
16853 |
18369 |
| VC++ 6.0 |
505 |
1598 |
676 |
1659 |
79329 |
0 |
337 |
1517 |
| Java 1.3.1 |
1185 |
2192 |
1521 |
2246 |
10341 |
1017 |
1020 |
3364 |
Table 3: Listing of all the run average runtimes for each of the tests and languages. All times are presented in milliseconds.
|
Integer Addition |
Double Addition |
Integer Multiplication |
Double Multiplication |
Object Instantiation |
Object Property Get |
Object Property Set |
Object Method Call |
| Variability Min |
0.00% |
0.05% |
0.00% |
0.10% |
0.03% |
0.00% |
0.02% |
0.05% |
| Variability Max |
2.21% |
0.65% |
2.02% |
0.67% |
2.37% |
2.10% |
1.48% |
1.47% |
| Variability Average |
1.03% |
0.28% |
0.86% |
0.31% |
0.56% |
0.83% |
0.88% |
0.51% |
| Variability SD |
0.69% |
0.19% |
0.74% |
0.18% |
0.87% |
0.90% |
0.62% |
0.50% |
Table 4: Composite listing of the variability of the test results for each type of test. Notice that all variability is below 2.5%, and on average below 1.7%.
Graph 1: All the data, graphed. Please note, some data is way off the graph. It's displayed like that so that you can see the bars for all the data.
Graph 2: C# vs. Java
Graph 3: C# vs. VC++ 6.0 vs. Java
References
Other
All the code will be submitted as a tool to DevHood (hopefully it's approved). If you have any questions, please feel free to email me!
Credits
Visual Studio .NET, C#, Visual Basic, Visual C++ are copyrights of the Microsoft Corporation. Java is a copyright of the Sun Corporation. Performance test results are copyright Robert A. Wlodarczyk, 2002.
|
|