When developers start working at a fintech, they know the stakes are high, but not everyone has access to a crusty Cobol developer with a life’s worth of banking experience to avoid every pitfall. In my experience, working closely with about a dozen fintech businesses, no one building their products had previously worked on a core banking system. But if the products we build contain financial transaction information that needs to be audited or reconciled, then our systems share many requirements with core banking systems.

Given a lack of well audited, public open source software for doing banking, and a lack of “here’s how we did it” blogs on the internet from trusted companies like Stipe or Venmo, fintechs need to either hire experienced developers or risk learning some lessons with financial transactions the hard way.

In an effort to make some of these lessons less likely, I’ll enumerate nine common engineering mistakes early fintechs make. Some I’ve drawn from my own experience, and others have come from interviewing members of the fintech community.

1. Security (obviously)

Of course, when money is involved, security is important. Security is a journey, a priority, a process, not a Jira card to complete. That said, there is one thing that I highly recommend everyone pay attention to that I’ve known many developers to be unfamiliar with: the OWASP top ten. I won’t list them here, but these are the 10 most common web application security mistakes as listed by Open Web Application Security Project. The list is kept up to date yearly as practices and tools change.

2. Using floating point data types

Everything stored in computer memory is in binary. When floating point numbers, which in software means numbers that have a decimal point somewhere in them, are represented in binary, they can lose precision. For example ten cents (0.10) in binary is like the number one-third (0.333…) in base ten. It’s a number that goes on forever. It looks like this: 0.000110011001100110011001100110011001100110011001100110011...

If you use floating point numbers to represent currency, computers will have to round them, and when you add up enough money, the rounding errors will start to add up and become noticeable, even substantial. Instead, developers should use integers to represent money and count pennies. If money needs to be divided, and division isn’t even, the extra pennies should be apportioned according to well-understood business rules.

Midjourney rendering of “a rounding error compounded over time in a fantasy painting style.”

3. Updating transactions

Transactions like credit card payments have a lifecycle with many states—authorized, cleared, voided, returned, declined, etc. It can be tempting for web developers that are used to updating the state of objects in a database to query for an existing transaction and update it. This is against fintech rules because of the financial requirement for audibility. Auditors looking at a ledger need to trust that every entry in the ledger was made at a specific time and has not been manipulated in any way. Fintechs need to build transaction tracking systems that add every transaction (even updates to existing transactions) as new records and to lock down databases so that no one, not even god, can change an existing transaction.

Over the last decade, fintechs using RDBMS databases to record transactions have built features onto their databases like a journal of all changes to all data and clever ways of making sure the data hasn’t been modified by storing checksums of data as transactions are added.

Assuming that fast moving entrepreneurs would like to find a way to get to MVP faster than might be possible with the level of engineering required to make a rock solid, RDBMS ledger system from scratch, here are a few options.

I should note that Kelsus has not evaluated all of these, so this list does not represent my endorsement:

1. This method suggested by AWS to use their QLDB database looks quite interesting. QLDB comes with audibility, write once, and write optimized access built in. It’s worth noting that one fintech architect I spoke to immediately rejected the idea of using QLDB for core banking because it has built-in scalability limitations and doesn’t have reliable market history.

2. I love challenges to the common wisdom, and the common wisdom of fintech is that you shouldn’t use a NoSQL database to store transactions. This is because out-of-the-box these databases are configured to be “eventually consistent.” It’s fair to read “eventually consistent” as “possibly inconsistent at any given moment.” And if a database is inconsistent, someone could, for example, spend the same money more than once. But there are ways, via database configurations and application code, at your own risk, to make NoSQL databases verifiably consistent. If you know that you’re going to have internet scale transaction throughput demands and want to use DynamoDB, here’s an article with information about how: https://decimals.substack.com/p/things-i-wish-i-knew-before-building

3. If you want to build your own ledger on an RDBMS, here’s an article for that (but it doesn’t get into data protection and auditing validation): https://blog.journalize.io/posts/an-elegant-db-schema-for-double-entry-accounting/

4. If you want a managed service to just take care of this for you, have a look at Twisp.

5. And finally if you want a product to do this for you, but want to host it yourself because you are concerned about third party managed service providers, have a look at TigerBeetle.

4. Multitenancy:

If your B2B SaaS app for making charter cruise reservations has a weakness that allows someone from one charter company to enter the id of another charter company into the API and get their data, the worst that can happen is a little unfair charter boat competition. In a fintech, these kinds of multitenancy bugs can lead to money losses and privacy regulation violations. Historically, multitenancy in SaaS has been enforced through careful code review and testing, but data across tenants all went into the same database, and there wasn’t a structural way of preventing a SQL query from returning data across customers. Since 2016, major databases like PostgreSQL and Sequel Server have supported row level security thereby making it possible to make parts of the database invisible to people without permissions. If you use an RDBMS for your fintech, definitely consider it as a way of keeping customer data safe. Here’s an intro about how to use it from AWS.

5. Testing and release management

Deciding how to configure environments—ie dev, sandbox, prod, etc—and how to ensure they are running properly can be a brain teaser in any organization. In fintech, when real money moves on prod, and core functionality depends on numerous third party integrations, it is super challenging. This article is not the place to explain how to do it, but there are three common mistakes to avoid.

1. Many companies have a really confusing or poorly documented path from sandbox to production for third party integrators. Make this path clear, and while you’re at it, do a risk analysis on whether it could ever be possible for a third party to connect to the wrong environment without catching their own mistake.

2. Plan for every important configuration of your production environment to be testable. Testability might require multiple test merchants and test accounts that move real money. You never know when you need to double check that the plumbing is working.

3. An often overlooked requirement is to be able to ensure that the production environment is doing error handling properly. The thing about production systems is that they’re not supposed to have errors, so you can’t see how they performs under error conditions unless you can force errors to happen.

6. Application monitoring

Deploying to a public cloud like AWS means that some infrastructure monitoring can be added quickly and almost as an afterthought, but application monitoring cannot. Even on day one of your launch, the CEO will want to know things like how many transactions are being processed and what their total dollar amount is. Another thing that tends to happen early in the life of a fintech is that there will be a drop-off between signups and usage, and someone on the product team will start asking if users are experiencing errors. Be ready with good monitoring to be able to answer these kinds of questions from day one.

7. SDK and Library support

Kris Hansen, CTO of Synctera, a banking as a service (BaaS) company, gets credit for this one. A problem developers, especially those that love using new tools and frameworks, can run into, he said, is “not considering the SDKs and libraries that you will want to include in your project. Knowing how to wrap and use third party libraries for things like DOCv step up, remote deposit capture, and security features is pretty key these days. If your stack choices make this really challenging you're going to have a hard time.”

I can imagine, for example, some serverless architectures that have a lot of business logic in AWS Lambda functions might make it difficult to do dependency and release management with these types of libraries. It’s also worth considering whether the libraries and SDKs you are using are just wrappers for HTTP APIs. In that case, I recommend using caution. I’ve seen too many bugs in third party HTTP API wrapper libraries over the years to use them without first considering whether it might not be better to just call the HTTP API directly with your own code.

8. Be careful with retry logic

Talking with two time technical cofounder and current fractional fintech CTO Cho-Nan Tsai surfaced a mistake that can lead to a cascading failure. He says, “When integrating with various payment vendors, be sure to set up async queues for those tasks. However, take care with how retries get triggered and handled. We had issues with too many unnecessary retries that blew things up in our system.”

I love this one because it is earned experience. His team was being careful and thinking ahead to do retries on failed API calls, but if a critical system under heavy load isn’t responsive, those retries can pile up.

More generally, I’ve seen both in and out of fintech that unresponsive third party systems are a paved road to cascading system failures. I’d like to suggest that the best—better than code reviews, better than pair programming, better than automated tests—way to prevent getting caught off guard is to do what AWS calls a Game Day. It’s essentially just a risk analysis meeting where developers talk about what could happen when things start going wrong. If you don’t have it as part of your release process, add it. You’ll be amazed at the mistakes you avoid.

9. Loosely defined requirements

Doug Hurst, former Venmo Head of Growth, mentioned this common mistake that might seem like a product rather than an engineering issue, but it falls to the engineering team to avoid.

Engineering teams working outside of fintech may be used to getting product requirements in the form of mockups and high level business requirements. Senior engineers on their own or in informal meetings distill these requirements down to the level of business rules, which get baked into systems and sometimes forgotten.

Doug says, “having your developers write their own requirements,” is a recipe for unexpected issues and slow feature velocity, because “the product dev interface is so crucial.” He went on to say that if you’re at a fintech where this is happening, take the extra time to document the detailed business rules around transaction state management, error handling and retry logic, and account configuration. Schedule time with the product team, even if they are understaffed, to communicate these rules back out to the business.

I asked in a few places for other examples of tech mistakes that fintechs have made and I am sure that I’m leaving out some surprising and facepalm examples. Please reply to this message with some of your favorites, and I can include them in an update next week.

Thanks for reading. Please consider hiring our team at Kelus to do software development for your company. It goes without saying that we don’t need to be trained on these particular mistakes.

—Jon Christensen