Notes

Sending notifications to a person whose photo you liked: RabbitMQ -> Celery
Django / Python for web server / application
PostgreSQL to store users, medias, friendships, etc.
- Master with multiple replicas, where reads happen on replicas (Master-Slave Replication)
- To deal with increased latency with writes, by batching requests wherever possible
- Replication lag from Master to slave replicas was not a big issue (for them)
Cassandra NoSQL (wide column store) to store user feeds, activities, etc.
- All replicas have the same copy of data, with eventual consistency
Image conversion is done at read time, not upload time

Scaling

Used a pod or group of Django, RabbitMQ to represent the application stack for one region
- Problem #1: Initially memcached would be in each local pod, but that lead to data staleness (memcached data from pod #1 could be stale vs. pod #2). Couldn't use memcached across regions and keep global consistency
- Solution #1: Did cache invalidation. When a PostgreSQL insert comes in for pod #1, a daemon on the Postgres read replicas does cache invalidation in their local pods
- Problem #2: More database reads, especially for really simple stuff like number of user likes, which used to rely only on cache
- Solution #2: Indexed table, increased read speed by orders of magnitude.
- Problem #3: Still high load on DB though
- Solution #3: Use memcached lease-get, least-set. Good tradeoff for UX like 1 million likes vs. 1 million and 10 likes (some staleness is okay).
Server growth was outpacing user growth - not good
Switched from PostgreSQL to TaoDB to avoid long replication times
- TaoDB is a DB + Cache internal at Facebook, uses write-through architecture
- Simplified object model which makes development easier
Use Canary to load test in production (500's vs. 200's)
Deployments to 20,000 servers in about 10 minutes