📝 Abstract:
The Internet and the World Wide Web is a crucial service that we rely heavily on for our daily lives. Modern web applications carry immense importance through our reliance to it for access to information and communication. This importance goes hand in hand with the security of the Web. The most popular way of Web security research is through large scale web measurements. In these measurement studies, researchers crawl and access a set of the web from a set of vantage points using some configurations to infer the security of the servers and the privacy of the end users through their findings. Since the Web is extensive, web measurements use a set of seed pages, often through top-lists that consist of the most popular webpages according to some authority. Furthermore, the choice of the methods for accessing and crawling the webpages affect the results as well. In this work, we evaluate these methods, propose explainable seed set selections and crawling methods that reflect and respect the structure and the organization of the web. We are aiming to develop tools to investigate the state of the web security better instead of one-size-fits-all measurement techniques.