Abstract:

The paper presents the architecture concepts for building crawling cluster for data-driven on-page optimization tasks. It contains a detailed description of the major problems and a thorough introduction to the SEO process and its components. Moreover, the developed and tested architectural concepts have been presented that can process big data for on-page optimization using a highly scalable network of distributed crawlers utilizing the state of the art technologies.