In the process of rebranding we decided to switch our architecture from a Django monolith to microservices. We had 4 weeks to redesign, rebrand and rebuild our entire site and so needed to figure out how to do all of this the lean way. We carefully split our architecture decisions into thin vertical slices that could be implemented over time. In this post I’ll talk about the main challenges we focused on, how they were implemented and how we built a microservice architecture while still staying lean.
Before I go into depth about the process, here is a quick diagram of our architecture:
To get there, we split the process into 3 main tasks:
1. API Layer
The API layer acts as a gateway and is the entry point to our architecture. It is pretty dumb and, for now, only does 2 things:
- Route requests to services
- Authenticate users (not authorise)
Node.js and Express made sense for this as they are extremely lightweight and the team were all familiar with Javascript, speeding up development . For authentication we use JWT (JSON Web Tokens)*, becuase of its simplicity and JSON being native to Javascript. After the user is authenticated, the API then routes the request to the appropriate service. The services are considered to be in a trusted network and are accessed by a private token passed in the ‘Authorization’ header plus the user id of the requester in an ‘X-USER’ header. This stops us from duplicating any authentication code across services and separates concerns so that each component only has to worry about its own specific authorisation and permissions. In the future we may move to a private network but not wanting to add the overhead of devops and migrating off Heroku meant that private tokens sufficed.
**there are some great libraries to easily set this up. For example, we use https://github.com/auth0/node-jsonwebtoken*
2. Decoupled Frontend
Building the API layer was the first step in decoupling the frontend. The next was to move out of Django, which was currently serving our templates, to a self contained frontend which consumes the API. To do this we chose Angular.js. There are currently a lot of mixed emotions in the tech community when it comes to frontend JS frameworks, be it which one to choose or downright hatred of them all. We didn’t want to again be locked down in an all-encompassing framework, such as Django, which would defeat the point of changing the architecture in the first place. Saying that, lock-in was one of our biggest worries with Angular, as it has a particular way of doing things that ties you into its way of development. At the end of the day though there are very few frameworks that don’t lock you in to a certain extent.
Frontend JS frameworks represent the direction that the web is heading, providing a lighter more seamless experience. Until there are improvements with Web Components, Angular represents one of the best of the bunch. Backed by Google, it has a growing community, ongoing development and support and I suspect it is going to be around for the long haul. Even if it is not, or even if we find a fundamental flaw with its implementation, having decoupled the frontend it allows us to easily re-write without affecting the rest of our system. This separation also forced us to ‘dog food’ our API which is going to be used by third party integrations (such as the 3 Beards), instilling best practices, such as versioning and proper RESTful design, from the get go.
Our other two worries about Angular were SEO and browser support. SEO is an important factor for us (being a primarily listings oriented site). Luckily Google now crawls Javascript sites with no penalty and for everything else we use Prerender. Regarding browser support we had to make the decision to only support the last 2 versions. This wasn’t a big issue for us as the majority of our (current) users use the latest version of Safari, Chrome or Firefox. This’ll probably change over time as our user demographic expands, although, by definition, if we put this off long enough then the task will be finished without us having to do anything… Joking aside, with older browsers we make sure that they they don’t fail ugly and redirect appropriately avoiding cluttering of our error logs and an unpredictable user experience.
3. Exposed Monolith
Finally, how do we deal with our monolith? We decided to treat it as if it was a (very large) microservice. We exposed our endpoints using the Django Rest Framework and registered it and its auth token with the API layer. Most of our logic had been abstracted out of views and into model methods which limited the amount we had to rewrite, although despite that there is still a lot to refactor…
With this basic architecture we are slowly splitting out the functionality of the monolith at our leisure. So far this has lead to separate search, billing and messaging microservices and already reduced the size of the original repo by 50%. Our architecture is now split over 2 main languages, Python and Javascript, but we have the flexibility to add more as long as it makes technological and business sense. We’ve also recently added a message queue in the form of IronMQ and are utilising their IronWorkers for self-contained asynchronous tasks. We’ll see overtime how this affects service design going forward.
After all this, there is still lots to do. We’ve not implemented any caching, our API documentation is terrible and test coverage is sparse: at first it was a surprise when anything worked! But these are all things that can be fixed gradually over time. Often when you read online about the Netflix or Instagram architectures, the steps they took to get there get lost. What I wanted to share is those early steps on that journey and that you can architect a stripped down microservices system using Heroku, cloud hosted systems and no devops. As with all things, moving to a microservices architecture will not solve all your problems and it has to be done well. It has different challenges to a monolith and, being fairly new, these challenges tend to be less documented. We’ll see in time how our architecture evolves at Hubble. The beauty and curse of a marketplace is that there are so many moving pieces that I’m sure it is going it’ll be a hell of a lot different in 6 months time.
And don’t forget if you are looking for some awesome London office space you can check our website.
