1. Crawling:<\/p>\n

\n
• scan websites<\/li>\n
• analyze and parse web pages<\/li>\n
• detect and collect URLs links and web resources.<\/li>\n<\/ul>\n

  2. Download resources from web-servers using automatically collected or provided URLs including dynamic JS rendered web-pages and store them in a shard local raw file storage.<\/p>\n

  3. Processing of a web page with customizable applied algorithms like unstructured textual content scraping, statistical data mining, NLP data mining and so on<\/p>\n

  4. Store results in local SQL DB storage with distributed multi-host and multi-process architecture model.<\/p>\n

  5. Crawling, processing, and data archiving management.<\/p>\n

  6. Distributed data architecture tasks like aging, purging, statistical and more.<\/p>\n

  7. Tasks scheduling and balancing using tasks management service of multi-host architecture or real-time multi-threaded load-balancing client-server architecture.<\/p>\n