Topological investigations attempt to discern structure out of the basic elements of the architecture and the links between them. Structure can tell us a lot. The investigation of the structure of the Web is of course always dependent on the level of abstraction of its description. Such is the size of the Web that even very small differences in the performance of these components could make large differences at the macro level. For instance, though one would not generally be worried
by the difference between an algorithm of O(n) and an algorithm of
O(n log n) in most problem spaces, on the Web scale the log n term
could start to get appreciably large [191]. Hence the behaviour of the
components of large-scale networks is of relevance even when looking at the global properties of the Web.
Furthermore, structure in turn provides evidence of what conver- sations are taking place over the Web. Hence understanding structure is important for a number of applications, such as navigation, search, providing the resources to support online communities, or ameliorating the effects of sudden shifts in demand for information.
The Web is democratic to the extent that there is no centralisation or central coordination of linking. Conceived as a hypertext structure, its usability depends to a very large extent on effective linking; following a chain of badly linked pages leads to the well-known disorientation phenomenon of being ‘lost in hyperspace’. Following a chain of links is also rendered less risky by Web browsers which contain ‘back’ buttons, which in effect provide the inverse of any hyperlink. And navigation need not only be a leisurely amble around a chain of hyperlinks, thanks to search engines that find pages with characteristics of interest to the user.
Web topology contains more complexity than simple linear chains. In this section, we will discuss attempts to measure the global struc- ture of the Web, and how individual webpages fit into that context. Are there interesting representations that define or suggest important prop- erties? For example, might it be possible to map knowledge on the Web? Such a map might allow the possibility of understanding online com-
munities, or to engage in ‘plume tracing’ – following a meme, or idea,
or rumour, or factoid, or theory, from germination to fruition, or vice versa, by tracing the way it appears in various pages and their links [5]. Given such maps, one could imagine spotting problems such as Slashdot surges (the slowing down or closing of a website after a new and large population of users follow links to it from a popular website, as has happened from the site of the online magazine Slashdot) before they happen – or at least being able to intervene quickly enough to restore normal or acceptable service soon afterwards. Indeed, we might even discover whether the effects of Slashdot surges have declined thanks to the constant expansion of the Web, as has been argued recently [166]. Much writing about the Web seems to suggest that it is, in some ways, alive, evolving and out of control [e.g. 87], and the decentralised
model of the Web certainly promotes the view that its growth is beyond control. The Web-as-platform model means that there are genuine and powerful senses in which the “creators” of the Web (who can be con- ceived as: the early conceptualisers of pervasive links between knowl- edge and knowledge representations; the originators of the powerful standards and languages underlying the Web as we know it; the many professionals currently and selflessly undertaking the painstaking nego- tiations on W3C standards bodies; or the writers of the actual con- tent that we see online) do not control the macroscopic structure. This model is very powerful, but that does not mean that the Web has nec- essarily become an undifferentiated soup of connected pages.
Methods of analysing the web looking at patterns of links [171] have turned out to be remarkably interesting, illuminating and powerful in the structures they uncover. For instance, some sites seem to be taken as authoritative in some way – in other words, many other sites link into them. Other sites contain many links out – one way of conceiving this would be that such sites index authorities on some topic – and
these useful sites act as hubs. Such hubs may also be authorities, but
equally they may be pointed to by few pages or even no pages at all. When methods such as those pioneered by Kleinberg, Brin and Page take the link matrix of the Web and find the eigenvectors, it turns out that they correspond to clusters around the concepts that the pages are about. Such authority-hub structures are of immense importance to our understanding of the Web, and require analysis of the link matrix to find. Indeed, Kleinberg’s original intention was to discover authorities, and the ubiquity online of the more complex authority-hub structure was initially a surprise [171].
Several authorities on the same rough topic are likely to be pointed to by all or most of the hubs which specialise in the area. Hence even if the various authorities don’t point to each other (perhaps because of commercial rivalries), they are all still linked in a fairly tight sub-
network by the hubs. Such structures can be seen as defining ade facto
subject or topic, as created by an actual community of page authors. Such topics and communities are an alternative way of carving up the content of the Web along the lines of standard classificatory discourse [137].