It is necessary to have a reasonable data volume estimate to help size the server architecture. Architectures may well vary with the size of each indexed content item. Indexing web pages of a few kilobytes and a DMS server (Document Management System) with heavy PDFs is quite different.
For recommendation purpose, the following points can be considered:
- For 100,000 web pages, an average 10-GB back-end space is necessary to store the crawled pages and to generate the indexes. And 1 GB is necessary at the front-end level to store the generated indexes and the snippets.
- For 100,000 structured items (of eCommerce catalog entry or database item type), an average 5 GB back-end space is used for indexing, and 500 Mb for the index front-ends.
- For logs, an average 1 GB is necessary per layer of 5-million queries.
- A two-processor server of the latest generation makes it possible to index more than 1 million daily documents (according to the type and complexity of the applied processes).
To allow users to adjust their hardware equipment to the best usage of AFS v7.9, the Antidot recommendations include:
- 2 cores minimum of a physical CPU (recommended: 4)
- 4 GB minimum of memory
- An SSD
- 100 Mbps of network bandwidth
- An ethernet card
To run tests, it can be sufficient to install Fluid Topics on a single server.
The server size depends on the types of tests to run but the following sizing is a minimum:
- Data storage: 50 GB
- RAM: 20 GB