Understanding Why You Want to Use IronPython for Testing
Every testing technique you’ve ever used has some drawback. For example, if you include debug statements in your code, you must ensure that you perform a release build to remove the statements before you release the code. Otherwise, the application will run slowly. In addition, using debug statements can cause the application to perform differently from the way it performs when you use it in a production environment, which makes it possible that the very tests that you depend on to check the application will actually hide problems from view.
Using IronPython for testing has a considerable number of benefits over other testing tools. The biggest benefit is that you don’t have to do anything special to the application. The test harness you create exists outside the application and doesn’t affect the application in any way. All the test harness does is monitor application behavior and report on it to you. As a result, if the test harness reviews every aspect of the application and verifies that it runs correctly, the application will run correctly in the production environment, too, because nothing will have changed.
As you’ve seen throughout the book, IronPython is an interpreted environment. That means you don’t have to create the test harness in one piece — you can create it a little at a time as you try things out with the application. In fact, the very nature of IronPython makes it possible for you to play “what if” analysis on your application. You can see just how bad you can make the application environment and try things that users do, such as create unreasonable execution conditions.
Using an IronPython script for testing means that all the testing code is in one place. If you decide that you need to add another test, you don’t have to delve into the inner workings of the application to add it and then create another build. Instead of using this time-consuming process, you simply add a few more lines to an external script using any text editor that you like. There’s nothing complicated about the process — anyone knowledgeable about your application should be able to do it without any problem.
The external nature of IronPython also makes it impossible for your test code to add problems (such as errors, performance issues, or reliability concerns) to the application. In some cases, adding test code actually introduces an application error, making it hard to know whether the error is in the test harness or the application. If there’s a problem in the IronPython test harness, you’ll see an IronPython error telling you about it. In short, you have separation between the test harness and the application, which ensures one won’t affect the other.
There are a few downsides to working with IronPython as a testing tool. The most important of these issues is that IronPython treats your application like a series of black boxes. It provides input to a method and expects a certain output. However, IronPython can’t see into the method to test individual elements within it.
IronPython also can’t see private members of your application, so it can’t test absolutely every aspect of your application. If a private member is causing a problem, you need to use some other tools to find it. Of course, you can use IronPython to infer certain issues in private methods based on their effect on public methods, but this kind of logic can prove more troublesome than direct testing.
Considering the Test Environment
Before you begin writing your test harness, you need to consider the test environment. The test environment determines how you test the application, be it a DLL or a desktop application with user access. The following list provides some criteria you need to consider as part of the test environment.
- Code access: You must define how the test harness will access the code. It’s important to determine whether the harness will test absolutely every method, property, event, and other application element individually, whether it will test elements in combination, or whether it will use a combination of individual and combined tests.
- Test ranges: A test harness must test both the possible and the impossible. For example, you might design a method to accept positive numbers from 0 through 5. However, the test harness must also test numbers greater than 5 and less than 0. In addition, it must test unexpected input, such as a string.
- User emulation: When working with some applications, you must determine how to emulate user activity. For example, you might write down a series of steps that the user will take to perform a certain activity and then execute those steps in your test harness. Of course, users are unpredictable; your script must also perform some haphazard and unpredictable steps and provide unexpected input. If you find that users are doing something you never expected, you must add it to the test harness.
- Security testing: If you don’t try to break down the walls you erected for your application, someone else will most certainly sign up for the job. Because IronPython tends to treat everything as public, it actually makes a great tool for testing security. You’ll find no artificial walls to keep things neat and tidy. Security is never neat or tidy — it’s all about someone ripping away the veneer of the façade you called security when you put the application together. IronPython lets you test your application brutally, the same way someone else will.
- System characteristics: Even though you can’t write code to ensure that your application will run on every machine in the solar system, you can do things such as add random pauses in your code to mimic activity on an overloaded system. You can also execute your application and its test harness on a number of different machine configurations to verify that the application will run as expected.
There are probably other criteria that you need to consider for your individual testing scenario. Take time to brainstorm scenarios, worst-case situations, and truly horrifying events, and then test for them. The following sections provide some additional insights about the test environment and the issues you must consider.
Defining Access
The matter of access is an essential part of testing. The word “access” has all kinds of meanings and connotations. Of course, there’s the access of your test harness to the code within the application. The black box nature of IronPython prevents access in depth, but careful programming can provide access to unprecedented amounts of information within your application and make testing relatively complete.
You must also consider the access the user has to the application as part of the test harness. For example, if you use external configuration files, you can count on some number of users accessing them. Even if you work out methods that are seemingly impossible to overcome, a user or two will find a way to overcome them. Anything you can devise will be overcome by someone (it’s always easier to destroy than to create). Consequently, you must consider all forms of user access as part of your test harness — if for no other reason than to determine how bad things can get when a user meddles.
It’s also important to consider external access. Whenever a system has access to the network or the Internet, you must consider the potential for outside sources to access your application (even if your application isn’t designed for such access). Many vendors of shrink-wrapped software have gained notoriety for not taking this kind of access into consideration. The thought was that the application didn’t access the outside source, so there wasn’t any need to consider the outside source during testing. It turns out that any outside access opens avenues of influence and attack for all the applications on a system, so you must test this kind of access.
Access is a two-way street. As part of your testing harness, you must consider application access to external resources. For example, you must consider what happens when an application attempts to access a particular file on disk and can’t find it. Even more important, you need to consider resources on the network or on the Internet. There are many forms of access that your test harness must consider as it tests the various methods inside the application. It isn’t always possible to test simply for strict inputs or outputs; you must test inputs and outputs within the confines of an environment defined by various kinds of access.
Considering a Few Things IronPython Can’t Test
Earlier, you learned that IronPython tests application elements using a black box approach — given a particular input, what should the element provide as output? However, there are other limitations you need to consider in the way IronPython performs testing. For example, IronPython can’t perform stress testing. If you want to test your application in a memory-starved environment, then you need to combine IronPython with another tool. For example, you might want to read the article at http:// msdn.microsoft.com/magazine/cc163983.aspx about a load-generating tool you can build yourself. Web application load testing requires other techniques that you can learn about at http:// support.microsoft.com/kb/231282. If you need to stress test applications in combination with a particular server, check out the site at http://blogs.msdn.com/nickmac/archive/2004/10/06/ server-stress-tools.aspx.
IronPython can perform diagnostic testing of your application with ease, but it doesn’t make a good environment for performance testing. As with stress testing, you need to combine IronPython with another tool to check application performance in various conditions. In fact, you may very well need to combine IronPython, your stress testing tool, and your performance testing tool to obtain statistics for a range of test scenarios and environments.
The point of this section is that while IronPython is a good scripting tool or a good diagnostic tool, it can’t do everything. In many cases, you must combine IronPython with one or more additional tools to obtain the desired information about your application. Your test plan should include all of these contingencies, and you should consider them before you create your test harness.
Creating the Test Harness
An advantage to working with IronPython is that you need not create the test harness in one sitting. You can use an iterative technique to create the test harness. It’s possible to start with a small nugget of tests that you know you must perform, and then add to that nugget as other issues come to light. Eventually, you end up with a full-blown test suite.
Most .NET developers won’t initially understand the benefits of using an interpreter for testing, but the realization will grow with time that interpreters make things easy. If you get an idea, you don’t have to run a complete test or compile anything. All you need to do is open up the IronPython console, load the assembly you want to test, and then try out various tests until you come up with a perfect combination of items to use. At this point, you can click the system menu in the IronPython console, choose Edit ➪ Mark, highlight the text you want to copy from your experiments, and press Enter to copy it to the clipboard. Now you can paste the text you’ve created into your test harness and comment it. In fact, the IronPython console (and all consoles for that matter) provides a number of commands, as shown in Figure 18-1.
As an alternative, if you already have the beginnings of a test-harness check, but want to add to it, you can always paste the text directly into the IronPython console using the Paste command shown in Figure 18-1. The interpreter will automatically execute any statements that you paste into it, so you’ll be ready to start typing new code after you paste it.
Modularity is the name of the game when it comes to a test harness. Try to place the individual tests into separate files so that you can reuse the code later. Simply have a centralized file where you call each of the tests in turn. The tests will output the information you need to screen, so the developer using the test harness need not even know that there are multiple files involved.
Testing DLLs
DLLs present one of the easier ways to begin using IronPython to test applications. In fact, you’ve already performed a kind of testing in Chapters 16 and 17 when you created the extensions and then used their content as part of an IronPython application. All that a test harness will do is formalize the testing process so that the output you receive speaks directly about the functionality of the DLL under test, rather than its use in an application. The following sections describe how to perform a test on a DLL using IronPython.
Creating the Test DLL
The DLL used for testing purposes is extremely simple so that the section doesn’t focus more on an interesting DLL than it does on testing techniques. All that this DLL provides is an account where you make an initial deposit to create the account and then make deposits, withdrawals, and transfers. The DLL includes a number of features so that you can try things out, but the code definitely isn’t production quality. For one thing, most of the error-checking code is left out to keep the code clear so you can easily see what will happen next. Listing 18-1 shows the DLL code used for this example.
Listin g 18-1: Defining a DLL to test
[code]
public class Accounts
{
// Contains the current account amount.
private Int32 Total;
public Accounts()
{
// Sets a default acccount amount.
Total = 5000;
}
public Accounts(Int32 Initial)
{
// Set a user supplied initial amount.
Total = Initial;
}
// Provides access to the account total.
public Int32 GetTotal
{
get { return Total; }
}
// Adds a deposit to the account.
public Int32 Deposit
{
set { Total += value; }
}
// Subtracts a withdrawal.
public Int32 Withdrawal
{
set { Total -= value; }
}
public void Transfer(Accounts Account2)
{
// Place the money in the second account in the first account.
this.Total += Account2.Total;
// Withdraw the money from the second account.
Account2.Total = 0;
}
}
[/code]
The example includes two constructors (something you didn’t try in Chapters 16 or 17). The developer can create an account with a default value of 5000 or provide some other initial amount. In either case, you end up with a new Accounts object that has Total defined.
The GetTotal property is read-only and lets the developer obtain the total in the count from Total. Using a property enables you to perform checks before allowing people to have the information. For example, you could place a security code in this property to ensure that only authorized personnel received the information. If a developer were to take this approach, you’d need to write a test to check the GetTotal property using an account other than the developer account.
The Deposit and Withdrawal properties are write-only. The caller doesn’t receive anything back from them. You could use a method to perform the task as well. Using a property makes the test code easier to read, but that’s the only advantage. In both cases, the properties change the value of Total. Of course, you can perform checks in the properties, such as verifying that a withdrawal won’t result in an account with a value less than 0.
The Transfer() method moves all the money from one account to the other. Typically, you’d provide some type of transaction support in a method of this type, but the example doesn’t include it. This is one situation where IronPython can test the method’s inputs and outputs, but can’t test the internal workings of the method. You’d need another tool to test issues such as whether the transaction support actually worked as intended.
Creating the DLL Test Script
It’s time to build an IronPython script to test the DLL shown in Listing 18-1. In this case, the test script is a bit short and doesn’t test every contingency (such as starting with a negative amount in the account), but it demonstrates how you’d create a test script for a DLL. Listing 18-2 contains the code needed for this example.
Listin g 18-2: Developing a DLL test harness
[code]
# Creates a new heading.
def CreateHeading(Test, Title):
print ‘n########################################‘
print ‘Test ID = ‘, Test
print ‘Test Title = ‘, Title
# Displays the values.
def ShowValues(Expected, Received):
print ‘Expected Value = ‘, Expected
print ‘Received Value = ‘, Received
if Expected == Received:
print ‘Test Passed’
else:
print ‘Test Failed’
# Ends the test.
def CreateFooter():
print ‘########################################‘
# Print out statements of everything the test is doing.
print ‘Beginning Test’
print ‘Loading clr’
import clr
print ‘Loading test module’
clr.AddReferenceToFile(‘TestDLL.DLL’)
from TestDLL import *
CreateHeading(‘0001’, ‘Creating Account1’)
Account1 = Accounts()
ShowValues(5000, Account1.GetTotal)
CreateFooter()
CreateHeading(‘0002’, ‘Making a Deposit’)
Account1.Deposit = 1000
ShowValues(6000, Account1.GetTotal)
CreateFooter()
CreateHeading(‘0003’, ‘Making a Withdrawal’)
Account1.Withdrawal = 500
ShowValues(5500, Account1.GetTotal)
CreateFooter()
CreateHeading(‘0004’, ‘Creating Account2’)
Account2 = Accounts(3000)
ShowValues(3000, Account2.GetTotal)
CreateFooter()
CreateHeading(‘0005’, ‘Transferring Money’)
Account1.Transfer(Account2)
print ‘nAccount1 = 8500’
ShowValues(8500, Account1.GetTotal)
print ‘nAccount2 = 0’
ShowValues(0, Account2.GetTotal)
CreateFooter()
# Pause after the debug session.
raw_input(‘nPress any key to continue…’)
[/code]
Let’s begin with the three functions at the beginning of the script: CreateHeading(), ShowValues(), and CreateFooter(). It may seem a bit silly at first to create these functions, but they provide a method for changing the output of the tests quickly, should you need to do so. In addition, you don’t want to write the same print statements hundreds of times as you create your script. It’s far easier to simply call the functions.
The CreateHeading() and CreateFooter() functions don’t have much logic in them — they simply display information onscreen. The ShowValues() function does have a bit of logic. In this case, it simply compares the expected value to the result and displays the appropriate output text. However, you could perform any number of checks required by your application. For example, if you’re working with strings, you might need to check the string length and determine precisely how it differs from another string.
Notice that the __main__() code begins with print ‘Loading clr‘. It’s important to describe every event that occurs in the test script. Otherwise, you won’t know where a script has failed during testing. Make sure you describe the mundane acts of loading and unloading modules, as well as the actual tests.
The first test begins with a call to CreateHeading() with the test number and title. The code then performs a test, Account1 = Accounts() in this case, calls ShowValues() to test the result, and finishes with CreateFooter(). Almost all of the tests follow the same pattern.
The final test is a little different than the rest. To perform the test correctly, you must evaluate the content of both Account1 and Account2. This is a case where you can infer what is happening inside a method with the test code. The method, Transfer(), could perform the task correctly with Account1, but not with Account2, which would tell you something about the content of the method and where to look for the problem.
This final bit of script also shows the flexibility of using the three functions presented earlier. By separating the individual tasks into three parts, you can call the ShowValues() function multiple times as needed. You might also consider creating a second form of ShowValues() to accept a comparison string for output (the print ‘nAccount1 = 8500‘ part of the script).
Performing the DLL Test
It’s time to run the DLL test. If you configured your project using the techniques in Chapters 16 and 17, you should be able to click Start Debugging (or press F5) to start the build process. During the build process, the compiler checks your DLL for major errors.
After the DLL is built, the IronPython script runs. Remember that this script is running outside of the IDE, so nothing it does will actually affect the performance of your code. The diagnostic tests will run and provide the information shown in Figure 18-2.
Notice that the use of formatting, test numbers, titles, comparison values, and so on makes the test results extremely easy to read. Of course, a large DLL could overwhelm the capacity of the console to display information. In this case, you could just as easily send the output to a text file, HTML page, or an XML file. The point is that the script makes it possible to view diagnostics about your application almost immediately after you build it.
Testing Applications
You can use IronPython for more than DLL testing — you can also use it to test your applications. Applications are more of a challenge than DLLs because you have to find a way to emulate user input. Of course, many developers just aren’t as creative as users. A developer would never think about putting text where a number is expected. Many developers discover, to their chagrin, that users will also try implanting scripts and doing other weird things to the application that aren’t easy to test. Some users will even try odd character combinations looking for hidden application features or just to see what will happen. Tests will only work as well as your ability to outguess the user. The following sections show how to test a simple Windows Forms application.
Creating the Test Application
The test application is very simple, but it does include some internal code you can use for testing purposes. The following sections describe the test application.
Defining the Form
A Windows Forms application need not be complex to test it using IronPython. All you really need are a few controls and some buttons with code for their event handlers. Figure 18-3 shows the simple form used for this example.
As with Windows Forms you use in a DLL, you must make an important change to test an application using IronPython. All the controls you want to access must have their Modifiers property set to Public. The default setting of Private prevents you from accessing them directly in IronPython.
Building the Code
You can see that the form in Figure 18-3 has three Button controls in it. Each of the controls has a Click() event handler associated with it, as shown in Listing 18-3.
Listin g 18-3: Defining an application to test
[code]
private void btnQuit_Click(object sender, EventArgs e)
{
Close();
}
public void btnAdd_Click(object sender, EventArgs e)
{
txtResult.Text = (Int32.Parse(txtValue1.Text) +
Int32.Parse(txtValue2.Text)).ToString();
}
public void btnSubtract_Click(object sender, EventArgs e)
{
txtResult.Text = (Int32.Parse(txtValue1.Text) –
Int32.Parse(txtValue2.Text)).ToString();
}
[/code]
The btnQuit_Click() event handler is as you might expect. It simply closes the form using the Close() method. You won’t test this functionality using the IronPython script.
The btnAdd_Click() event handler converts the values of txtValue1.Text and txtValue2.Text to Int32 values using Int32.Parse(). It then adds the numbers together, converts the result to a string using ToString(), and places it into txtResult.Text. Because IronPython needs to test this event handler, the visibility is set to public. If you don’t change the visibility of the event handler, IronPython won’t be able to access it. The btnSubtract_Click() event handler works the same as the btnAdd_Click() event handler, except that it subtracts the two numbers.
Creating the Application Test Script
As long as you’re willing to make the required visibility changes to your application, you can use IronPython to test it. Creating a test project for an application works precisely the same as creating a test project for a DLL. Here’s the short list of changes you must perform:
- Change the build output location for both the Debug and Release builds to the solution folder.
- Add IPY.EXE as an existing project to your solution.
- Set the ipy project as the startup project so that the IDE executes it instead of the Windows Forms application.
- Configure the ipy project to start your script and to use the appropriate working directory.
- Add a new IronPython script to the solution folder.
This test script uses the three functions described in Listing 18-2 to provide output. It also adds the following two output functions:
[code]
# Verify the type.
def CheckType(Object, Type):
if Object.GetType().__str__() == Type:
print ‘Test Passed’
else:
print ‘Test Failed’
# Show initial values.
def ShowInit(Value1, Value2):
print ‘Value1: ‘, Value1
print ‘Value2: ‘, Value2
[/code]
The CheckType() function compares the type of an object you create against an expected type. If the type is incorrect, then it displays a failed message. You can use this function when creating a form or other object that could fail for any number of reasons.
The ShowInit() function displays the initial values for a binary operation or perhaps just two values used for some other task. You could probably create a version of the function that accepts any number of arguments in the form of an array. The point is that you can create some specialized functions to display data for a particular test and then find that you can use it for other purposes later.
As previously mentioned, this test script also uses the three functions found in Listing 18-2. Listing 18-4 shows the actual test script for this application. It doesn’t provide a complete test but does provide enough information that you could easily complete it if you wanted.
Listin g 18-4: Developing an application test harness
[code]
# Print out statements of everything the test is doing.
print ‘Beginning Test’
print ‘Loading clr’
import clr
print ‘Loading System assembly support’
import System
print ‘Creating a blank event argument.’
EventArg = System.EventArgs()
print ‘Loading test module’
clr.AddReferenceToFile(‘TestApplication.EXE’)
from TestApplication import *
CreateHeading(‘0001’, ‘Creating a test form’)
MyForm = Form1()
CheckType(MyForm, ‘TestApplication.Form1’)
CreateFooter()
CreateHeading(‘0002’, ‘Testing a default add’)
MyForm.btnAdd_Click(object, EventArg)
ShowInit(MyForm.txtValue1.Text, MyForm.txtValue2.Text)
ShowValues(‘2’, MyForm.txtResult.Text)
CreateFooter()
CreateHeading(‘0003’, ‘Testing a default subtract’)
MyForm.btnSubtract_Click(object, EventArg)
ShowInit(MyForm.txtValue1.Text, MyForm.txtValue2.Text)
ShowValues(‘0’, MyForm.txtResult.Text)
CreateFooter()
CreateHeading(‘0004’, ‘Testing add with one change’)
MyForm.txtValue1.Text = ‘5’
MyForm.btnAdd_Click(object, EventArg)
ShowInit(MyForm.txtValue1.Text, MyForm.txtValue2.Text)
ShowValues(‘6’, MyForm.txtResult.Text)
CreateFooter()
# Pause after the debug session.
raw_input(‘nPress any key to continue…’)
[/code]
The test script begins by loading the required code for the test, beginning with clr. Because this test has to work with event handlers, it needs to load the System assembly and create a System.EventArgs object, EventArg. Because the event handlers in this application don’t actually use the event arguments, EventArg is actually a default object with no content. The call simply won’t succeed without it, however, so you must create it.
After the script finishes the prerequisites, it performs the first test, which is to create the Windows Forms object, Form1, as MyForm. The creation process could fail; you want to verify that MyForm isn’t null, so that’s the first test that relies on the CheckType() function. You don’t have to show the form to test it, so the code doesn’t call ShowDialog(). If you do decide to show the form, you’ll actually need someone to work with it. The script is suspended during the time the form appears onscreen.
The next step is to perform some tasks with the form. The code performs a default add and subtract. The two value fields, MyForm.txtValue1.Text and MyForm.txtValue2.Text, contain default values that you can use for testing. Actually, it’s good application design to always include default values for the user so that the user has some idea of what kind of content to provide.
The MyForm.btnAdd_Click() and MyForm.btnSubtract_Click() event handlers perform the actual addition and subtraction. In order to call these two methods, you must supply both a sender object and event arguments. The sender object can simply be an object because the code doesn’t use it.
The final test in the example is to change one of the values and perform another addition. To perform this task, the script changes the value of MyForm.txtValue1.Text and calls MyForm.btnAdd_Click(). Normally, you’d provide a wealth of additional tests to check various values and see how they react with the code. For example, you might provide some negative values to ensure that the event handlers work properly with them. You might also test incorrect input, such as providing a string. The point is that you can completely automate any level of testing using this IronPython script technique.
Performing the Application Test
At this point, you have an application to test and the script to test it. It’s time to run the application. One of the problems you could encounter is not making something public (such as an object, control, or property) that you need to test (the default is to create private objects, controls, and properties). Unfortunately, the need to make class members public is one of the problems of using IronPython for desktop application testing. It’s not a big problem, but you need to consider it. When working with an extremely large application, changing the required member visibility could prove problematic. In addition, making some members public could pose security risks.
Let’s hope everything works as anticipated when you run the test. Figure 18-4 shows typical output from this application.
As with the DLL testing script, this script outputs text that’s easy to read and results that are easy to decipher. You know immediately whether certain tests have failed and precisely what inputs were used to conduct the test. As with DLL testing, you may need to use some other form of output, such as an XML file, when performing testing on complex applications because the content won’t fit entirely onscreen.
Performing Command Line Tests
For many developers, testing must be formal or it really isn’t testing. Actually, ad hoc testing is sometimes better because you get to play with the application while you test it. Testing in an ad hoc manner at the command line is possible in IronPython because it’s an interpreted environment. In fact, we’ve been performing ad hoc testing throughout the book. Every time you reviewed the content of an application, no matter what type it was, using the dir() function, you were performing a kind of ad hoc testing because you were reviewing the content of the application.
The test demonstrated that the DLL had added the overrides correctly and that you should be able to access them from an application. In addition, you discovered that IronPython views the content of the DLL in a slightly different manner than another environment might view them.
Let’s look at a specific test example, the TestDLL.DLL file. For the purposes of this example, you want to use the dir() function to determine whether the Accounts class contains everything it should (and nothing it shouldn’t), as shown in Figure 18-5. Notice that there’s no mention of Total in this list, but you can see all of the properties and methods described in Listing 18-1.
If you remember from Chapters 16 and 17, the __doc__() function is undefined for an assembly that you import into IronPython, but the help() function does produce a result. One of the next checks you should perform manually is to verify that the assembly provides the kind of information you expect from help. Figure 18-6 shows the output of the help() function for the Accounts class. Notice that it contains all of the information you expect, including the fact that there are two forms of __new__(), the constructor, and the read/write state of the various properties.
Of course, you’ll want to perform other sorts of manual testing that could eventually appear in your test script. For example, you might decide to check whether the Accounts class will let you create an account with a negative starting amount (it will).
It would be also helpful to know whether someone could circumvent some of the properties in the Accounts class. You wouldn’t want someone to use code such as Account2 = Account2 + 20 to overcome the protections in the Deposit property. In this case, the test displays an error. Another check might include adding two accounts together, such as Acccount3 = Account1 + Account2.
By now, you should have the point of using manual testing. You can creatively think of ways that someone might try to overcome protections in your code. It probably isn’t possible to find every avenue of entry into a DLL, but testing in this way helps you think through more potential problems that other forms of testing allow. Interactively probing your code is a unique method of testing the impossible.
