The basic principle of crawling consists in considering a web page and then extract links of it to find new links that will be followed, aso.
Without restrictions, crawling can obviously be boundless and thus endless. To avoid this kind of inconvenience, many restrictions must be set