There can be no doubt that Osgiliath experienced a period of growth at least in the early centuries of Gondor's existence, and the city even if confined to the Bridge originally must have flowed out onto both shores of Anduin. The river was quite wide, perhaps a mile wide, at the city's location, and the Bridge would have contained many houses and towers. Its population should have numbered in the thousands, and perhaps fleets of ships sailed under it as far north as Cair Andros. Of all the works of the Numenoreans in Middle-earth, the Bridge of Osgiliath must have been the greatest.
The Twin Fortresses of Aglarond and Angrenost
There has been much debate concerning the purpose and origins of these two fortresses. At best we know that they were built by men of Numenorean race and that they figured prominently in the wars of the later Third Age. There is little enough written about the fortresses. In "Of The Rings Of Power And The Third Age" Tolkien says:
...The chief city of [Gondor] was Osgiliath, through the midst of which the Great River flowed;
and the Numenoreans built there a great bridge, upon which there were towers and houses of stone wonderful to behold, and tall ships came up out of the sea to the quays of the city. Other strong places they built also upon either hand: Minas Ithil, the Tower of the Rising Moon, eastward upon a shoulder of the Mountains of Shadow as a threat to Mordor; and to the westward Minas Anor, the Tower of the Setting Sun, at the feet of Mount Mindolluin, as a shield against the wild men of the dales. In Minas Ithil was the house of Isildur, and in Minas Anor the house of Anarion, but they shared the realm between them and their thrones were set side by side in the Great Hall of Osgiliath. These were the chief dwellings of the Numenoreans in Gondor, but other works marvellous and strong they built in the land in the days of their power, at the Argonath, and at Aglarond, and at Erech; and in the circle of Angrenost, which Men called Isengard, they made the Pinnacle of Orthanc of unbreakable stone.
"The days of their power" could mean quite a few things. The Numenoreans assembled their greatest armies, Elrond tells us, during the days of the Last Alliance of Elves and Men. And yet Gondor reached the peak of its power during the days of Atanatar Alcarin, who lived more than 1,000 years after the overthrow of Sauron by the Last Alliance. It's unlikely that Tolkien envisioned the "days of their power" extending past the Gondorians' retreat from Mordor in 1636. The Great Plague so devastated their realm that they were no longer able to maintain the watch upon Sauron's realm afterward. And yet, Gondor weakened itself during the Kin-strife, which occurred some 200 years previously.
We know that Tarannon Falastur extended Gondor's control westward along the coasts of Middle-earth. He must have conquered the Anfalas, probably Andrast, and perhaps even Enedwaith. Angrenost and Aglarond would have provided him with valuable bases in that time, but they could also have been built to help control the Gap of Calenardhon. So it's probable that the fortresses were not built any later than circa TA 913 (the end of Falastur's reign). After his day the Ship-kings were concerned with taking Umbar and other parts of the Harad, so it's unlikely they would have needed any fortresses in the northern lands. In fact, since Falastur was himself the first of the Ship-kings, we may guess that he conducted his conquests from the Sea rather than by land.
So then what strategic value would either Angrenost or Aglarond have held for the Numenoreans
"in the days of their power?" From Gondor's founding until the overthrow of Sauron Mordor was indeed a threat to the western lands, but Appendix A in THE LORD OF THE RINGS says that Elendil and his sons believed "that Sauron also had perished" in the ruin of Numenor, so the threat they would have perceived was not one of Sauron's return, but rather of the presence of those of his servants who had survived his death in Numenor.
However, if Angrenost and Aglarond were intended to defend Calenardhon against Mordor they were poorly positioned to do so. An enemy army could cross the entire province before reaching the fortresses. Anduin would have been a better place to defend the region against attacks from the east, and in the Third Age the river was indeed fortified for just such a purpose. It's hardly likely that the Gondorians possessed a military lore superior to that of the Numenoreans.
Yet Calenardhon was a part of early Gondor. We know this from the survey of the realm Isildur is said to have made in The Tradition Of Isildur, given in the chapter "Cirion and Eorl" in UNFINISHED TALES:
It is said that when Isildur returned from the War of the Last Alliance he remained for a time in Gondor, ordering the realm and instructing Meneldil his nephew, before he himself departed to take up the kingship of Arnor. With Meneldil and a company of trusted friends he made a journey about the borders of all the lands to which Gondor laid claim; and as they were returning from the northern bound to Anorien they came to the high hill that was then called Eilenaer but was afterwards called Amon Anwar 'Hill of Awe'. That was near to the centre of the lands of Gondor....
Isildur must have been in Calenardhon if he returned to Anorien from the north. We also know that Sauron had at one time positioned an army in the lands east of Calenardhon, for he is said to have burned the Entwives out of those regions, creating the Brown Lands, in an effort to slow the advance of the Host of Gil-galad and Elendil. Could Anarion have built the fortresses at that time? It seems unlikely, as they still would have served no purpose and the construction would have required many men and much time, of which Anarion is said to have had too few to withstand Sauron in any event.
It seems logical that the Gwathuirim of Enedwaith were the reason for the construction of these fortresses. These people were related to the Men of Dunharrow, the "wild men of the dales" in the lands to the south, and to other men who had accepted Elendil's rule in Arnor. For whatever reason Enedwaith was not incorporated into either Arnor or Gondor and its peoples, hostile to the Numenoreans for centuries, were probably enemies of both realms.
Thus, it would be logical for Isildur and Anarion to have built the fortresses during the first century their realm existed. The presence of a Numenorean force in Aglarond would have helped defend Calenardhon against the Men of Dunharrow (although it could only have represented a threat to them from the flank), and perhaps was one reason that Isildur was able to conclude a peace with them. When they broke their oath to march with Gondor in war, the fortress would have helped convince them not to attack the Numenoreans from the rear.
But what was so wondrous about these two fortresses? The fortifications themselves were not extensive. Angrenost had its circle but the great wall of Aglarond was built late in the Third Age by the Rohirrim. Orthanc was a wonder in its construction, being tall and so well built that even the earth-crumbling power of the Ents could not harm it. Isildur and Anarion (or their heirs) placed a Palantir in Orthanc, using the fortress to watch over the northern bounds of their realm.
Aglarond really only had the glittering caves to mark it as something special. But it is perhaps significant that Gondor (and Rohan) never mined the caves despite the obviously rich mineral deposits there. The natural beauty of the caverns may not have been a shrine, but nonetheless must have inspired a sense of awe akin to that which Gimli and Legolas expressed when they saw the caverns for the first time. The Numenoreans may indeed have appreciated the special beauty of the caverns and elected never to alter them for fear of destroying one of the great wonders of Middle-earth.
Taken together, Angrenost and Aglarond represented a vivid memory of the power of Numenor.
Even at the end of the Third Age they provided visitors with a majestic and sensational view even though the great power which raised them had long since retreated south beyond the White Mountains.