More information about the Underscore mailing list

[_] Database choice for a social media platform

Erik Wallace erik.wallace at lewdewe.com
Wed May 18 18:48:09 BST 2016

Hi _ folks,


I'm researching technical details for a proposed social media platform similar to what Twitter et al do. I see that the big boys are using Hadoop for their main archive storage and search. Some articles on the matter suggest using a smaller scale system such as MongoDB etc. near the front-end to store data on the most recent activity and then have Hadoop further back for long-term data. However, I have no prior experience of Hadoop so I don't know if I'm barking up the wrong tree or biting off more than I can chew.


Part of me thinks it would be best to aim high and not have to redo everything later, however, I realise that a new platform would start off small, and in fact may never gain substantial traction, so I could just stick with a medium-weight database such as MongoDB, or at a pinch maybe MySQL, and then hire development assistance to convert it all to Hadoop if it ever became big enough to warrant that?


If anyone has opinions on these technologies or what points I should be considering then I'd welcome some feedback. Thank you!



Kind Regards,

Erik Wallace