This guest post is written by Oyvind Henriksen, CEO and co-founder of Poq – the commerce platform for native apps. Poq became part of the Seedcamp family in 2012 and are currently raising funding on Seedrs as part of a new £500k round.
One of the things we learned when working with large retailers, is that they always have problems with their integrations. They have too many legacy IT systems, and they generally don’t talk well with one another. This means the information we need is not available in one place, which makes implementation projects a pain. Our product won’t work unless it’s connected to the right datasources.
One of the great benefits of Seedcamp was getting in touch with other startups with similar challenges. We learned about scaling our database from Nuji, and about combining data sources from Lyst. I hope this post could be as helpful for the rest of the community.
How we made integrations a part of our product
We decided to tackle the integration challenge head-on, and build integration into our platform. Instead of requiring the retailer to copy everything into our system, we can now access all the information we need directly from their systems, in real-time.
One of our clients require information from 9 different IT systems to run, and each system often has hundreds of thousands of records. Using our new integration architecture, we were able to go from signed to live in 11 weeks.
Here’s how we did it…
We got the idea one day when we were parsing through 5 files of 2 gigabytes each, looking for ways to synchronise all that data into our database. We then decided there must be a better way, so we shifted our thinking. What if we could leave the data out of our database in the first place? What if we could treat the API as our database, for real-time purposes? And that’s when our real-time API was born.
Turning our platform inside-out
The centre of our platform used to be a database, with a CMS and an API. We’d then synchronise our central database from the client’s exported feed. This is the old-school architecture that’s simple, reliable and proven to work.
Now, the centre node of our platform is not a database, but a hub. The hub decides in real-time where to get different pieces of data. We also distributed our systems to embrace a more Service Oriented Architecture. So when a consumer on a retailer’s app taps on the dresses category, our hub will ask the e-commerce platform for the products, then look up the corresponding store stock, prices, product reviews, pictures etc from all the different systems it knows about. It then merges the data together, and sends it to the app for display.
Making it fast enough
We always talk about sub-second response times, because we believe that faster systems are the best way to increase conversion rates.
After this change, our platform still had a sub-second response time. Our average response time for the last million API requests is today 177ms, the median is 41ms, and 95% of requests return after about 600ms. We’re also maintaining our 99.95% uptime since the change, so we’re not seeing the client’s APIs go down too often.
Our first method to improve performance was putting a Redis cache between all systems. The app-facing API has a cache for every request, and each integration point is cached. The dedicated caches also improved scalability, which came in handy during the recent Black Friday and Cyber Monday retail events!
We then created an algorithm to merge the data in the fastest way possible. We use a multi-threaded and asynchronous architecture, and managed to both run the network calls simultaneously and also merging the data using all of the server’s CPUs at the same time.
Building the other side of integrations
Most retailers of a certain size will have a product feed with the product information. The files are usually CSV or XML format, and are updated one or more times per day. Our first construction on the other side of the integration was creating our own format.
Our first integration guide had only 2 sections: Field names for the product information, and querystring names for the shopping cart transfer. We then had clients follow this guide, and since they implemented the integration with our format, we did zero development work to get them live on the platform!
Our second construction was a set of plugins; for Magento, Shopify and Demandware. These pre-made integrations follow our integration guide, and can be easily installed and configured on the client side. We’ve now expanded our plugins to support APIs for essential functions like Native Checkout and Apple Pay, so there’s exciting times ahead!
We’re now seeing more and more APIs among our clients, so we think the age of feeds is finally drawing to an end.
If you’re offered a feed when working with a retailer, why don’t you ask them if they can expose an API for you instead? Because enterprises are embracing more and more of Service Oriented Architecture, they often have APIs available. And that might make both you and your client’s lives easier!
Your product does not stand alone!
The main takeaway I’d like you to get from this, is that your product does not work alone. The connection between your product and your client’s systems should not be done as one-off projects, you should build the integration as part of your product. That helped us a lot, and I hope it can help other SaaS products too!
Please feel free to contact me if you’d like to hear more about our platform. I always love to talk about our technology, and I’m happy to offer advice and opinion to anyone experiencing the same challenges we’ve had over the years.
If you’d like to learn more about our app commerce platform, check out poqstudio.com