Architecture at Web scale: the good, the bad and the ugly

Level : Intermediate

Have you ever wondered how large sites manage their architecture and technology when they have 100's of deployed services and a large, distributed engineering team? At Skyscanner we have both in abundance and have encountered many challenges as we have grown.

Our technology stack is varied and includes: C#, .Net, Python, Java, JVM, JS, Node.JS, Varnish, HAProxy, Nginx, IIS, Kafka, Samza, Docker, AWS, MySQL, Postgresql, SQL Server, DynamoDB, ElasticSearch, CouchBase, RabbitMQ…

Our system architectures (there are many) have fractal like properties with ever increasing complexity the closer you look…

How do we make sense of this complex and ever changing landscape? What problems have we encountered? How have we solved some of them?

This talk will provide you with insight into what works at scale and what does not from an architecture perspective. Sometimes it is a technology problem sometimes it is not but at the end of this session you will have some practical guidance that you can apply to your systems and organisation to help them scale.

In this session we will discuss the following topics and more:

* Evolutionary vs planned architectures
* Just enough architecture to keep your future self happy
* The importance of contracts and standards
* Technology adoption and why being opinionated is important as you scale
* Identifying and dealing with points of pain as you scale
* The top 10 areas that you should invest in early and frequently as you scale

 architecture  complexity  organisation  systems  devops  microservices  cloud  leadership

Paul Gillespie

I have worked in the software industry for 19 years on systems as diverse as energy supply management, SMS gaming (yes really!), mobile game content management, e-Commerce, daily deals and travel search. I am currently an architect at Skyscanner where I work in our Developer Enablement tribe where we try to remove pain points from our engineering organisation in areas such as deployment, micro-services, distributed systems and AWS.

@bigpg