Yet again, I’ve been fortunate enough to catch up with some fascinating developers (you could almost say “Interesting Developers“) from Bandtrace. They have 600-1000 unique visitors per day (and they are just starting up).
Is it just me, or are all Swedish people gorgeous looking?
So let’s find out how they are using Elastx powered by Jelastic and look under the hood to find out more about the challenges they faced, their current set-up and the brag-worthy results achieved with our platform!
Thanks for your time Tommy and Nathalie. So, what exactly is Bandtrace?
Bandtrace is a music encyclopedia that contains everything you ever wanted to know about artists: who they’ve been involved with, what they have released, what tracks they have written, recorded or produced, the latest news, etc.
Bandtrace can be described with one word: simplicity
It aims to be a comprehensive but yet an extremely simple-to-use website. Bandtrace has been carefully designed to contain all the data you need – no more – no less.
Who are the people behind the project?
Development of Bandtrace.com started two years ago. We are both web developers based in Stockholm with a strong background in Java.
Tell us about your biggest challenge
The three main challenges when building a mash-up with 20.000.000+ pages and 100.000.000+ links were:
1. The application architecture
A traditional solution backed by a relational database is not a perfect match for graph-like data.
2. To be able to fit loads of information on a mobile device
A healthy information structure is crucial and a mobile first approach is probably the only way to go
3. Hosting a cost-effective scale-up and scale-out solution
The runtime solution consists of several processes, NoSQL servers, search servers etc. This requires a flexible solution that can grow with the number of users. In addition to that our batch jobs crunches huge amounts of data every day – we needed to find a solution that had the capacity to crunch it.
What was the solution?
Jelastic offers both scale-up and scale-out alternatives. When the load increases the Jelastic platform simply scales up. When the number of servers isn’t enough it is easy to scale out and add more servers. Furthermore, the scale up solution is a perfect match for our daily batch jobs that process a massive dataset. We can complete the jobs in a few hours and then the server simply goes to rest, which is obviously extremely cost effective.
Elastx uses SSD based storage which is a perfect match for our Neo4j graph database. The graph containing the music data contains more than 100.000.000 relationships and it is not cost effective to keep the entire dataset in memory. Therefore, fast disk access is required to get a significant performance boost for the backend.
Can you share your set-up with us?
We mainly use the VPS-nodes as Jelastic does not support production-ready Neo4j or Jetty 9 yet (the good news is that they will be available via cartridges in upcoming releases). Below is an image showing the deployment.
The app consists of two parts – the web application and the batch application.
- Web application
- NGINX is used for load balancing, cache and web front.
- Jetty 9 with Java 8
- Database – Neo4j
- Search server – Apache Solr
- Batch application
- Standalone Java 8 application
- PostgreSQL (source database)
- Redis (cache)
- Neo4j (target database)
- Batch node – only used for importing new data once per day. 126 cloudlets (this is typically 0 cloudlets and then during import it scales to 110 cloudlets) – scale up
- Web nodes – 2 nodes to handle failover of 30 cloudlets (they never reach more than 20) – scale out
What are the benefits and results of using Elastx?
The benefits of a Java based solution using NGINX (web server and load balancer), Jetty (application server), Neo4j for back-end storage, Redis for cache, Solr for search indexes are:
Daily batches take approximately 3hrs to run on the Jelastic/Elastx platform. Buying a “fixed price per month” server on a different hosting provider for the same price led to batch jobs taking more than 24 hrs to complete.
- Our Neo4j queries are approximately 25% faster when read from SSD
- Roughly a 30% price decrease due to auto-scaling for the web application
- Possibility to deploy standard java apps makes development simpler. No vendor lock-in and no weird deployment scripts!
- 24hrs vs 3hrs for batch jobs (when using the “same priced server”)
In addition to that, when using Jelastic with Elastx we found a high performance solution that had enough capacity to run our gigantic batch jobs.
WOW! So you’re saving 21 hours per day in batch jobs, paying 30% less and queries are 25% faster?
Lastly, why Elastx powered by Jelastic?
When we decided on the Elastx solution we benchmarked the autoscaling against solutions that offers a “fixed price per month” and we simply got better performance and “more bang for the buck”.
Thanks Tommy and Nathalie!
Are Tommy and Nathalie The Most Interesting Developers in the World?
If you think they are, then nominate them here.