Back To Blog

1.9 Million Records Processed in 24 Hours

1.9 Million Records Processed in 24 Hours

After an exciting start to the year, Brolly has hit an important milestone! We’re now processing 60,000 records per day on average and processed over 1.9 million records in our busiest 24 hours.

How did we do it?

Brolly collects data through a ‘firehose’ in the native apps and then stores it locally (in Sydney, Australia) using AWS and S3. We use a distributed, stateless environment that moves data through a series of pipelines (queues). This allows us to easily scale horizontally by increasing the number of workers in our cluster, which happens automatically through the help of CloudWatch alarms and various custom metrics that we’ve hooked up through our system.

When we process social media records, we use ElasticSearch to ensure that the data is searchable and then we push it through a three tired, enrichment approach to provide safe, usable and exportable data.

First, the data is classified to ensure it can be displayed in a unified feed and Brolly can archive the content in real time, then we capture screenshots of all records to ensure Brolly is the most complete, records management solution. Multiply this by 1.9 million records and you’ll understand why we’re so thrilled by our new milestone. We know that social channels are growing every day, to test how much data our system could handle initially we even connected large public accounts (like Justin Bieber’s)!

What’s next?

We’ve been bombarded by interest from across government and the private sector and are excited to announce that we will be showcasing Brolly at the National Archives of Australia in Canberra on 31 August 2016.

Thank You

Every start-up has its story and Brolly was created out of the need to protect Australians and public organisations having important conversations on social media.

More than ever, people are making life changing decisions based on the information available on social channels. This could be as serious as a bushfire or flood alert, or as simple as a personal conversation with a trusted public provider.

We know that managing social media archival in a compliant and simple way is a conversation all public Australian organisations need to be having and we thank you for being part of the journey so far.

We’re working closely with our early adopters to refine key product features such as sophisticated reporting, integrating new services and refining direct messages.

Want to get involved? Contact us at hello@brolly.com.au