Testing of Software for User Behavior Compliance Management

Summary

A policy-based security audit and compliance software product was acquired by a major technology vendor and integrated into its IT service management suite, which required migrating the product from the Oracle database engine to DB2, moving user management to the new platform's directory server, and adopting new standards and documentation. INNERLUXES delivered full-cycle testing of the migrated release — heavy on automation — to ensure no functionality, performance, or security regressions.

About the Client

The Client is a vendor of policy-based security audit and compliance software. Its compliance product captures comprehensive log data, correlates it through sophisticated log interpretation, and communicates results through a dashboard for full audit and compliance reporting. To reduce the threats posed by privileged insiders, the product monitors change management procedures, acceptable use policies, and user authorization processes against company and regulatory policies. The product was later acquired by a major technology vendor and integrated into its IT service management suite, adding data governance and compliance monitoring, auditing, and reporting across mainframe and distributed environments.

The Challenge

After the product became part of a larger environment, a complex integration task arose. To comply with the new environment's standards, the product had to be moved from the Oracle database engine to DB2, and its user management — earlier based on internal authentication — had to move to the new platform's directory server authentication. All project documentation had to be recreated in the new format. Moving to the new environment also meant new standards of work, including new documentation templates and tooling. The team began using more detailed specifications and based their work on the quality assurance plan as the main test document.

The dedicated team had special assignments: a system test in an environment as close to the Client's as possible, an extended performance test to ensure the database migration wouldn't degrade performance, and an internationalization test for the product's multiple languages. With development spread across countries, communication complexity increased, and the team size was nearly doubled at the Client's request, with all testing performed by INNERLUXES.

The Solution

All test activities were coordinated with the international teams through weekly progress meetings and email.

Automated testing and unit testing

The main change in the release was the conversion to a different database platform, which posed risks to full functionality, as the migration affected all features. INNERLUXES put as much effort as possible into test automation — many tests were regression tests and ideal automation candidates. The most experienced testers created a test API covering the most complex functions, while the rest of the team (mostly manual testers) built automated test cases using that API. This improved the team's automation skills, increased automation coverage, and tested the product.

One of the most challenging tasks was testing the database conversion results — a set of complex views, stored procedures, and tables had to be converted from Oracle to DB2. To test the migration, the team created unit tests in PL/SQL in the Oracle environment and handed them to the database conversion team, who used them to verify their code after conversion. This saved the conversion team time (they didn't have to study the product's functionality) and protected the product from database regression issues.

Performance testing

An important requirement was that performance with the new database engine be the same or better. The challenge was measuring performance for the whole system, not just one aspect. INNERLUXES created a performance test council of testing specialists, development specialists, and performance analysts, which produced a performance measurement model and the necessary tools.

Manual functional testing

Despite good automated coverage, INNERLUXES still invested heavily in manual functional testing, since it was crucial that no functionality broke after migration. Several approaches helped finish on time:

• Reuse of prior test cases — using test cases from previous releases avoided designing from scratch and ensured they were correct and had sufficient test data.
• Test combination — engineers testing different parts combined their tests to share the same environment and data, using the output of some tests as input for others; this optimization saved about 25% of test time.
• Risk-based prioritization — risk analysis of the test plans prioritized scenarios by the impact and likelihood of failures, allowing lower-priority tests to be postponed if the plan slipped.

Security testing

The new release had to comply with the vendor's security standards, and its security had to be at the same or a better level than before. Because user management was recreated from scratch — and the product itself is a security system where the cost of any issue is very high — INNERLUXES, with developers and architects, created a security map identifying potentially vulnerable areas, addressing not only unauthorized access but also data integrity and protection of the strictly confidential data stored in the product.

The Results

• All test activities were performed on a regular basis, with test planning tracked transparently for the Client's management and intermediate deadlines met on time.
• Despite the different environments, communication with all development teams went smoothly, and code was tested quickly right after development finished.

Technologies and Tools

High-level GUI tests: IBM Rational Functional Tester, IBM Rational Test Manager, Selenium, AutoIt.

Middle tests for business logic & performance: Java, Perl, C/C++, Windows Script Host.

Database unit tests: PL/SQL.