Dropbox sync11/29/2023 ![]() File size: Brotli can use unicode or binary context bytes to dynamically select H uffman tables during runtime, which makes a multi-percent difference in final file size.Our initial research into Brotli was promising, and we identified 5 key advantages. While Dropbox had done research into generic file compression algorithms as well as Lepton, a novel image recompression algorithm, these techniques did not suit themselves to operating at network speeds on client machines. We had previously been using zlib for years but our measurements indicated we could do better with newer algorithms. We will also touch upon how our new investments can help us in the future. We will talk about the principles we kept in mind and what we have learned along the way. All together, these improvements have reduced median latencies by more than 30% and decreased data on the wire by the same amount. Today, we will dive into the Broccoli encoding, provide an overview of the block sync protocol, and explain how we are using them in conjunction to optimize block data transfers. ![]() ![]() To enable these advancements, we measured several common lossless compression algorithms on the incoming data stream at Dropbox including 7zip, zstd, zlib, and Brotli, and we chose to use a slightly modified Brotli encoder, we call Broccoli, to compress files before syncing. Thus, compressing files before syncing them means less data on the wire (decreased bandwidth usage and latency for the end user!) and storing smaller pieces in the back end (increased storage cost savings!). Modern compression techniques identify and persist these redundancies and patterns using short bit codes, to transfer large files as smaller, losslessly compressed, files. Because the files themselves are being sent to and from Dropbox during sync, we can leverage redundancies and recurring patterns in common file formats to send files more tersely and consequentially improve performance. We also constantly strive to better leverage our infrastructure, increasing Dropbox’s operational efficiency. It is vital that we constantly improve the sync experience for our users, to increase our users’ productivity in their everyday lives. Dropbox syncs petabytes of data every day across millions of desktop clients.
0 Comments
Leave a Reply.AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |