TiDB – The Agile Database for Fast Growth Fintech | GWEI 2022-Singapore

On July 14, the first “GWEI 2022 – Global Web3 Eco Innovation Summit – Singapore” was held at the Marina Bay Sands Convention and Exhibition Centre in Singapore, co-organized by Singapore’s Singapore University of Social Sciences (SUSS) and SUSS Node for Inclusive Financial Technologies (SUSS NiFT), and 8BTC’s new international brand “DeFiDAONews”.

The summit is sponsored by OKX, the world’s leading Web3.0 ecosystem builder.

Shawn Ma, CTO for APAC region from PingCAP, gave a speech titled “TiDB: The Agile Database for Fast Growth Fintech” to introduce PingCAP’s one and only focus – TiDB Cloud.

The speech content is edited by DeFiDAONews as follows:

PingCAP is a global company with 600 employees all across the world. The HQ of the APAC region is in Singapore. Our company focuses on building TiDB, an open source distributed SQL database. TiDB has already been  adopted by around 2000 adopters across the globe, with many big names included.

Here are some of the adopters’ logos and they come from different industries. Most adopters are from Fintech, banks, and insurance industries. Some are internet companies, logistic companies and SaaS.

Why TiDB?

Why do people use TiDB and what are the problems we’re trying to solve? First of all, a lot of Fintech companies are growing fast, accumulating data and users at a very fast speed. That’s one of the characteristics for Fintech companies.

Another problem is about decision-making, which tends to be in real time. Not only your operational team needs to make decisions based on real time data. But also, sometimes you put your real time data insights for your end users to attract them and to let them spend more time on your platform.

Meanwhile, risk control is very crucial for Fintech companies. You need to make business decisions, detect fraud and risk in real time. You also need to be very cautious about selecting solutions because Fintech is all about financial data, which translates into money. Therefore you cannot just choose random solutions. You need products that have already been proved in other Fintech companies,  traditional banks, or insurance companies.

Fintech is more agile compared to traditional banks. Traditional banks may spend years to  develop a new business, while it only takes Fintech companies months or even weeks to do the same.  When the data grows beyond the boundary of a single node database,  it’s hard to scale your mission-critical applications, and everything becomes super complicated.

When your data is of medium or small size, you can put everything in one server. This one server can solve all your problems and support millions of users. But once the data or your users grow beyond that boundary, suddenly you will find you need more databases or more servers. One database is simple, but with more databases, everything becomes a lot more difficult, especially if you are doing something like transactions.

For example, suppose you are transferring money from your account to another account. These two accounts might stay in two servers. What if one server goes down in the middle of the transaction? What will happen is that $10 will be deducted from your account, but not added to the other account, so the total balance will be broken. That’s something you want to avoid. If your mission-critical Fintech applications are running on multiple servers, things become extra complicated.

That’s one of the challenges users are facing. Another is, if you are trying to capture real time insights as data volume increases, the latency becomes higher and higher. If you’re trying to deal with a large volume of data, you will probably need a lot more skillful engineers and need to build more complex applications. That’s something you also want to avoid, as you  want to roll out your new business in weeks or months, not in years.

Our product, TiDB, is an open source database.  You can check the source code on github. It’s developed and maintained by contributors from all over the world. Its storage component, TiKV, is now a Cloud Native Computing Foundation, or CNCF graduated project. CNCF is a very famous foundation in the cloud computing area. We contribute a lot to the foundation, even more than a lot of big names.

Learning TiDB: Business Value, Research and Compliance Advantages

This is the architecture of TiDB. Firstly, it’s a distributed system so that we can deal with extra large volumes of data and workloads.TiDB is not only a distributed system, but also a hybrid system. What does that mean? It means TiDB is equipped with dual engines. One engine is called TiKV, which is the transitional engine capable of dealing with millions of transactions per second. The other part is the analytical engine called TiFlash, which is capable of dealing with a large amount of reporting on top of massive volumes of operational data to give you insights in seconds or in sub-seconds.

Nowadays everything is provided on the cloud. So is TiDB. You can directly use TiDB on the public cloud as a fully managed service. It provides you with a very easy and agile experience. You can just click some buttons and the service will roll out, and you can directly use the service to manage your data.

What are the business values enabled by TiDB? Previously, you probably needed to do data sharding or use some complex technology to manage your data. Now, with a simplified architecture for data management, TiDB enables more agile development and faster go-to-market processes.

We provide a very simplified architecture for you to manage data. No matter if your data is in medium scale or it is your plan to grow it into a larger scale, or you already have a very large scale volume of data – it can help you because it can be elastic. You can add or reduce  nodes from it. It’s suitable for a large range of data management. Also, while doing the scalability, you don’t face any downtime, because it’s very elastic.

Some of our users in e-commerce companies have a promotion season. They can add nodes before the promotion season, and after that, take the nodes out from the cluster. That will save your cost, and you can still support the peak hours with on-demand resources.

As described before, whenever the data is written into the database, you can directly analyze on top of your current data. It’s not like analyzing based on the history data, but on the current data.

How to describe the current data? Every time we release a new version, we will do some tests on the analytical engine. We will do simulated concurrent banking transfer, with millions of bank transfer transactions that happen on the cluster. At the same time, we will do the reconciliation test on the other side of the analytical engine. This means that even if you are doing analytical work on top of TiDB data, it guarantees you consistency. So it’s suitable for dealing with something related to financial data and money.

TiDB not only has influence in the Fintech industry, but has also contributed a lot to the research area. Two years ago, we published an introduction paper called TiDB: A Raft-based HTAP Database. It has been published on VLDB, a top tier conference in the database area.

Also we have given a lot of tech sharings in different international technical conferences. We have cooperations with prestige universities such as University of Wisconsin-Madison, ETH Zürich, RMIT,  CMU, and in China, Tsinghua University.

The Fintech industry might be very concerned about security and compliance, and the TiDB cloud actually provides very good compliance. We have SOC 2, ISO27001 and 27701 compliance, and also GDPR for the European customers.

Key Scenarios for TiDB

TiDB is a hybrid database. One of the use cases that TiDB provides is for the mission-critical workloads. You can use TiDB for online mission-critical workloads. It can be payments, or money transfers. You can use TiDB to easily scale the workloads.

Suppose that previously you had a few users and one day you find your user number or workload has increased beyond your current infrastructure capacity. In this case you can use TiDB to smoothly deal with more users and more data. Inside TiDB you can achieve high horizontal scalability, and that’s something we provide from day one. Also by the built-in design, every data has multiple copies and different replications. The different replications guarantee you the high availability of data. Even if some of the machines go down, the business continuity will still be insured.

And also it has a strong consistency compared to a lot of traditional distributed systems. Traditional systems do not provide you with global consistency. If you scale to more nodes, you need to take care of some of the consistency issues, especially for cross node transactions. Something disastrous might happen and the consistency will be lost. But that will not happen on TiDB, because we have a consensus algorithm called Raft built inside to ensure the data are consistent.

It also provides you with MySQL compatibility, which means that your previous data stack on the MySQL ecosystem can be translated to TiDB very smoothly without changing the code. On the other side, we provide building real time insights, which means that you can analyze all your data on top of the current one. And also, this will provide both your operational team or your end users with a lot of benefits to have the insights in real time.

Meet TiDB Users

Next are some of the use cases that we have. All the use cases I have chosen are in the fields of Fintech and Web3. Firstly, TiDB is used in a leading e-payment company in Japan. They are using TiDB for their core payment services. Before us they used another service provider, and one day they found their user number had exceeded 30 million, and the previous database had reached its limitation. Consequently their end users had to suffer much longer response time. Therefore they chose to use TiDB instead in their core payment services.

By making this replacement, they switched from a single node database to a distributed database in only 3 months. That’s a very short time for core payment services in large e-payment companies. The switch could be achieved in a short time because TiDB provides MySQL compatibility. Not much needs to be changed to adopt TiDB and no sharding is required, which means that you can have the similar experience as a single node database and no limitation like a sharding database.

Another use case, Square Cash APP, one of the most popular e-payment APP in the United States, had a similar situation. The biggest difference is that the previous customer was at the point to select new database solutions, and while they could have used sharding databases, they chose TiDB.

But in the case of Square Cash APP, they had already chosen the sharding database as their initial solution, but found it very hard to maintain. Then they switched to TiDB for easier maintenance, low latency, scalability and transparency. The reason they chose TiDB was similar – MySQL compatibility, asset transaction across different nodes and global index, no downtime during the scale, and good technical support.

I would also like to add some Web3 cases as well. For some reason, I cannot mention the customer’s name, but they are a big player in the Web3 world. They use TiDB to store the business operational data from the users. Other players who use TiDB are also big ones for the Web3 industry. They gathered the data from the public chain and store in TiDB in real time, so they can provide the end users with the public chain insights. The reason they chose TiDB is similar as well.

We encourage you to try TiDB cloud today, and it will empower the Fintech and Web3 industry with agility and simplicity. Thank you very much.