Understanding of Network Protocol Testing

Protocol testing can be grouped in two categories: Stress & Reliability Tests and Functional Tests.

• Stress and Reliability Tests include Performance Testing, Load Testing, Stress Testing, Line Speed Testing and Robustness or Security Testing.
• Functional Tests include Negative Testing, Inopportune Testing, Conformance/Compliance Testing, Syntax and Semantic Testing, Interoperability Testing and Deep-path Testing.

Performance Testing - process of verifying that the performance of the device under test meets an acceptable level. Performance testing is a superset of line speed testing in which performance applies to many aspects of a network device or application, and not just line speed.

Load Testing: Load testing is used to determine how well a system will perform in a typical environment under a specific load. Load testing is useful when you are confident that the application is functionally sound, but you do not know how the application will perform under a specific load.

The example of a load test is a call generator application that generates 1000 calls that a telephone switch must process. Does the switch continue to operate? Does its performance degrade or it crashes?

Stress Testing – process of subjecting the device under test to out of boundary conditions. The device under test should report an error message, gracefully shut down, or reject the input in some other way. In no case, should the device under test crash, reboot, or cause any harm.

An example of stress test can be 50 SNMP managers simultaneously querying one SNMP agent in the device under test. Normally, no more than one to three managers concurrently query an SNMP agent in a device. The device should perform an appropriate action.

Line Speed Testing - process of verifying that a device can operate at its rated line speed, when the bandwidth is 100% utilized or saturated.

For example, if the device is rated as operating at 5 Giga bytes per second, then the device should be able to handle incoming traffic utilizing all the available bandwidth.

Robustness Testing - process of subjecting a device under test to particular input streams. The input streams may be one of three types:

• Random input streams
• Valid input streams
• Invalid input streams

Robustness testing is a form of security testing. Security testing is more broadly defined to include monitoring / surveillance and the detection of specific exploits like IP spoofing or phishing etc.

The example of intelligent robustness test can be adding a trailing dot to the DNS name in the SIP URL. This is legal, but perhaps unexpected. The SIP implementation in the receiving device should process this correctly.

Negative Testing – process to verify that the device under test responds correctly to error conditions or unacceptable input conditions. Negative testing can be challenging because the number of incorrect conditions is unlimited.

Example of a negative test would be using a security protocol for authentication with an incorrect parameter.

Inopportune Testing – process to verify that the device under test is able to react properly when an unexpected protocol event occurs. The event is syntactically correct, but occurs when not expected. Inopportune testing is a specific instance of negative protocol conformance testing.

An example of inopportune testing is a BYE response to a SIP INVITE. The SIP INVITE is expecting a 100 Trying response, but not a BYE response.

Protocol Conformance Testing - process of systematically selecting each requirement in a standards document and then testing to see if the device under test operates according to that requirement. This is done by creating a series of single function tests for each requirement, resulting in thousands of tests.

Conformance testing for computer networking protocols is defined in ISO/IEC 9646-1:1994(E) as "testing both the capabilities and behavior of an implementation, and checking what is observed against the conformance requirements in the relevant International Standards."

An example of a conformance test is to check if the “ping” command operates correctly.

Syntax and Semantic Testing: Protocol conformance testing requires testing both the syntax and the semantics (functionality) of the device under test. Semantic tests force the device under test into a certain condition or state. Often the test cannot verify the correct behavior; it must be verified by an operator.

It’s example can be a test for a router to check whether it is maintaining an accurate count of all erroneous incoming packets of a certain type requires a mechanism for generating the erroneous packets, counting them, directing them to the router, assuring they were received by the router, and then reading the actual counter in the router.

Interoperability Testing - process of testing devices from multiple manufacturers by interacting in such a manner as to exercise the network protocol under test.

Interoperability testing is very useful in the early stages of a new product or new protocol. As products mature, interoperability testing becomes less valuable as it does not uncover enough new bugs to warrant the cost of setting up, configuring and managing network equipment from various manufacturers.

Deep-path testing – process of exercising every path through the code, not just the main path through the code. This is done by maintaining and tracking the protocol conversation very precisely, in a controlled way.

Deep-path testing is useful for forcing a particular path through code that is very difficult to exercise but very important for correct operation.