Shopee Data Event Center

Imagine that you're working on an awesome backend project. It's well defined and well decoupled from the rest of the subsystems in the company. At some point in time however, another team asks you to send them a stream some of the data that your project owns. This request doesn't really fit your project definition, implementation or API, but this data must be shared as per product requirement. So, you add a seemingly random, out of place piece of code that does exactly that, sends a very specific stream of data to some external system.

With time your backend project adds more and more such "additional" data streaming pieces of code, bloating the project and making it hard to track and manage. You can’t wait to get rid of it, but don't really know where to move this code...

Now imagine, a few days later, your project manager tells you to expect peak load due to a massive marketing campaign starting soon. As a responsible developer you run stress tests and find the bottleneck. You can fix it by moving some locking queries out of a transaction and processing them asynchronously somewhere else. But where?...

DEC to the rescue!

DEC (Data Event Center) is Shopee’s new programmable data pipeline. It is a new platform that allows asynchronous tracking of database events and triggering actions upon those events using built-in scripting engine. This platform solves the problem of streaming data changes from databases to any given destinations, be it another database, queue or k/v storage, with the ability to modify the stream of data that flows from source DB to destination sinks. The applications range from DB replication tasks, to async DB updates processing, DB to cache(s) synchronisation or generating task queues based on DB changes.

This talk will touch on the basic concepts and architecture that power DEC, and how it’s used in Shopee to solve these problems.

Speaker: Rim Zaydullin

Rim is a senior expert software engineer at Shopee and has been working on infrastructure projects in the internet industry for the last 12 years. He’s particularly passionate about data storage systems of all flavors, from in-memory only storage to distributed filesystems. Some of his previous work includes:

- Yandex Cocaine Cloud (https://techcrunch.com/2013/10/16/search-engine-giant-yandex-launches-cocaine-a-cloud-service-to-compete-with-google-app-engine/)
- CocCoc Bigdata project (https://bigdata.coccoc.com/en)

Comments

Post a Comment

Popular posts from this blog

Setting up Terminal in Android Studio for Flutter and Bash Profile (For Mac)

Does your Commands in Android Studio not worked? Fret not, all you need to do is the following from Command Prompt: 1. Do you have a Bash Profile?] Go into your  user folder in finder . The .bash_profile file should be findable there. ->  HD/Users/[USERNAME] To reveal hidden file, press: Command + Shift + . nano .bashprofile or nano .bash_profile 2. Paste the correct directory export PATH=~/ Users/rigmiklos/Documents/ flutter/bin:$PATH Red Text: Directory contain flutter folder Hint for Mac User: Double tap on the Flutter.exe in Flutter>Bin>Flutter, and press"Get Info" to get Directory. 3. Ctrl + X to exit (No, this is not for default paste function) 4. Press Y to save changes. 5. To reload source .bashprofile Advance Reading for Bash Profile Official Doc for Setting up To navigate around Mac Command prompt: ls: show folders cd "folder name": to change directory Solution: Run Flutter Doctor manually or by Terminal / Users/...
Read more

Flutter Widget 101 Part 1 (Basic of Flutters, Layouts, Rows and Columns)

Image
1. What every programme starts with void main() => runApp( new MyApp()); "=>" is an operator and it  denotes   { return  expr ; } . () denotes method runApp will run MyApp() which is a widget. Besides running MyApp(), you can also play with runApp() function by doing the following: new Text(). Do remember to include the TextDirection. Text will appear on the top left hand corner of the screen. void main() { runApp( new Text( 'Hello World' ,textDirection: TextDirection. ltr ,)); } Make it centralised in the center through the following. void main() => runApp( new Center( child: new Text( 'Hello World' ,textDirection: TextDirection. ltr ,) ) ); 2.Declaring App/Widget to be run in runApp() void main() => runApp(MyApp()); class MyApp extends StatelessWidget{ @override Widget build(BuildContext context) { // TODO: implement build return null ; } } extends is the inheriting of...
Read more

Flutter Widget 101 Part 4 (Navigation and Drawers)

1. Drawer Official Doc Material Design has 2 mode of navigation, Tabs or/and Drawer. Drawer slide in horizontally Use ListView (Scrollable) or Column as Child ListTitle for each single row Details of User can be displayed with DrawerHeader , UserAccountsDrawerHeader onTap Behaviour can be set Wrapper: Material App > Scaffold > Drawer > ListView Drawer: When tapped on row, will change Text in Main Page void main(){ runApp( new MaterialApp(home: new DrawerTemp()),); } class DrawerTemp extends StatefulWidget { @override _DrawerTempState createState() => _DrawerTempState(); } class _DrawerTempState extends State<DrawerTemp> { String TextValue = "Original" ; @override Widget build(BuildContext context) { return Scaffold( appBar: new AppBar( title: new Text( "Drawer Test" ), backgroundColor: Colors. lightBlueAccent , ), drawer: new Drawer( child: new ListView( ...
Read more