The cloud: scalable, standardized, smart
Knowing that an online intelligence solution needed to be as automated as possible from the outset, Sidekick architected the solution to scale massively via AWS services. The concept began to come to fruition when Sidekick developers created a solution to scrape html from websites and capture that data within a database. With 20 processes running per server, scaling up to 200 servers at a time, Sidekick was able to scrap the sites of over 140 Million websites in two days.
This data was then put through processes to glean standard pieces of information: name, addresses, product names, product details, categories, services, clients, client logo’s, etc. This information became structured and created the dataset for an interactive, organic, experience.