The challenges of performance testing in FinTech

Keeping up with the pace of today's exploding digital environment of diverse, interconnected tools and platforms is not an easy task - on-premises, cloud, legacy, SaaS, desktop, mobile. That's where performance testing and tuning comes in handy, testing systems' limits, finding bottlenecks and drawing up solutions.

• Running an IT department today is like conducting a non-stop music piece with an orchestra where, while performing, new musicians keep showing up bringing new, never-heard-of instruments all the time.

This can be especially demanding in fintech, where strong security, control and regulations are shaping the business. Let us look at some of the main challenges it creates while implementing performance testing strategies in the finance organizations.

Challenge 1: Security

Security is the mantra of any fintech solution and trust is the primary decision criteria. No client will use a financial service without a strong confidence that her private data are safe and professionally taken care of. Any sign of mistrust is a show stopper. As a result, multiple levels of security are being implemented to finance-related apps and systems.

Multiple-factor authentications

Such demands, in turn, make testing the performance of fintech systems extremely complex. When you replace a simple username/password procedure with two-or-more-factor authentication, tests have to emulate realistically additional actions like giving a finger print, receiving a security code via text messages or inserting a card with a certificate into a card reader. Certificates confirming users’ identity are all over the place, and the tests have to deal with that.

Custom plugins

Another particular factor is that many of these additional control and security mechanisms are custom-made, with transfer protocols and data formats used in non-standard proprietary ways. A performance testing tool thus has to be open and flexible enough to allow new plugins being quickly and easily made-to-measure. Here SmartMeter.io draws a significant advantage from its open-source Java core as compared to closed legacy systems with a limited set of callable functions. The full power of the Java language and ecosystem can be harnessed to create needed extensions as well as the vast resources created by the Java open-source community.

Deployment in various security zones

To properly test performance, one cannot allow for any unnatural latency that would not occur in the real production environment. The load generators thus must be deployed inside of the infrastructure in various high-sec or even demilitarized zones. Given that a clearance for root access is mostly tough to obtain, this can be a serious obstacle for performance testing tools that require executing a program. Here again, using platforms like Java helps. Load generators can be just unzipped and are ready to run.

Challenge 2: Variety

Multiplatform Environments

Another common challenge of performance testing in banks, insurance companies, and the likes is that they need to run on multiple heterogeneous environments. One system can have components spanning over several different operating systems, yet they need to be tested as a whole. Unix backend? Linux firewalls? Windows desktop client? iOS and Android mobile apps? Nothing unusual. That’s why the testing tool being platform independent is ever so important.

On-premises vs. Cloud

Fintech is a traditionally on-premises performance testing domain hiding from the world behind thick (fire)walls. While many systems remain inaccessible from the outside world, with the introduction of consumer-facing applications like internet and mobile banking, insurance quotes, automatic scoring, etc., these strongholds had to open up. Data exchange between internal systems and SaaS (systems as a service) platforms in the cloud is a standard today, and thus performance testing in fintech requires tools able to test from both the inside and the outside.

Keeping IT suppliers under control

Large corporations tend to have a very rich ICT ecosystems. For example, one of our European banking clients operates four core systems to which over 250 other systems and apps connect. Even if there was a general contractor for most of that, dozens of end-suppliers and vendors are usually in the picture. By testing the impact of any new or upgraded component of such ecosystem on the performance of the core systems, we can ensure a smooth operation once deployed to production.

Challenge 3: Coordination

Given the strict security and regulatory measures fintech is facing, a tight cooperation with all the institutions involved is a must. While preparing a load test, the testing experts need to get clearance to enter high-security parts of the infrastructure which can prove a rather difficult and lengthy process, stretching the limits of corporate hierarchies.

Working with authorities

Access and security clearances within the company itself are just one part of the equation while preparing a performance test. Often, you need to address also external authorities, either public or private, e.g. certification authority. Once, we needed to create 5.000 virtual users with real bank accounts mimicking customers interacting with the bank via an internet banking app. For this testing scenario, each virtual user needed to have a personal certificate assigned by a real institution - an official certification authority - to pass through a two-factor authentication into the system. Without previous experience with such a situation from their side, it took us and the bank weeks to negotiate and execute a plausible deal, both financially, and logistically. Only then we could move forward with the testing.

Scoring

Another typical coordination challenge is testing the performance of scenarios that require 3rd-party scoring of people or businesses. An example from our practice was testing a private loan approval system. The multiple types of loans, each with different application forms, combined with a lengthy 3rd-party scoring process of the borrower in question made it a rather complex task to come up with a meaningful way to test the system.

Testing data preparation

In standard e-commerce test cases involving e-shops and websites, creating a realistic dataset for testing is as easy as copying the production database with real user data and anonymizing them by altering the contents. In fintech, the data preparation task is a whole other animal. There can be so many interdependencies with other systems, cross-checks and limits in the fintech ecosystem, that the only way is to develop specialized robots and scripts that simulate the normal account creation procedure with all its steps.

This is exactly what we had to do in our previous example of testing the performance of an internet banking system with thousands of virtual banking customers. Simply copying and altering real customer data was out of the question and virtually impossible, and we ended up programming robots that would go through the whole opening account procedure like you would do at a retail office, including depositing money into the accounts. Our initial idea of simplifying that by making one account "a millionaire" who would then send money to all the other accounts didn't fly. With the transfer-per-day limits set by the bank/law regulations, it would take weeks.

Fraud detection systems (FDS)

Fraud being the most common malfeasance in the finance business, implementing 3rd-party fraud detection systems is a common practice to protect oneself and the clients. The systems use various approaches to detect a wrongdoing with sets of hard rules, soft rules, blacklists, whitelists, all the way to AI or machine learning techniques. The degree of seriousness of a possible fraud sets then off a corresponding escalation before allowing to complete the operation user intends to do. It can range from asking an additional security question ("What was the name of your first pet?"), through alerting a human operator to give the user a call ("Did you just book an airplane ticket from Suspicioustan?"), to blocking the account and alerting security.

For these fraud detection systems to work, they need to have all pieces of the puzzle together at any given moment. Massive amounts of data from all relevant systems have to pass through these "black boxes," creating an enormous pressure on their performance and making them the perfect candidate for thorough load testing. A performance issue on the FDS would bring the whole system to a halt, like malfunctioning traffic lights on a major intersection.

Now is the right time

As you can see, performance testing in fintech is not the usual test-thy-e-shop load testing on demand you buy from a cloud service for a few bucks. It requires a lot of knowledge and experience, as well as flexible, versatile tools that can be bend to cope with the diverse and high-security focused IT environments of financial institutions and companies. As complicated and demanding as it may be, making it an integral part of any fintech endeavor is not only a recommendable "best practice" but, as we believe in SmartMeter.io, a must.

The good news is there is a great window of opportunity opening right now. As IT departments turn towards DevOps and agile, bringing more automation and continuous integration into their daily work, implementing a comprehensive performance testing strategy is more feasible than ever. With the right tools and the right partner who will help you to design and implement the strategy and - I can't stress enough how important is this - trains your people. Performance testing has little value when not done regularly, so keeping all the knowledge with 3rd party experts leaving after the job will not bring you much further. What you should insist on is transferring as much of the know-how as possible to your people. That way the competence will grow in your organization creating the often missing ingredient of successful performance management - continuity.