Saturday, June 14, 2003

PTCL eats the humble PIE!



April 2003 proved to be an eventful month for the Internet community in Pakistan. Everybody - from occasional web surfers to power netizens felt the agonizing pain and frustration that was caused by the repeated, prolonged interruption in the Internet access on a nation-wide basis. For some people who knew the internals of the issue more than others, the whole series of event caused shame on top of all the pain.

The doomsday pundits and their warnings about such a worst-case scenario for the Pakistani Internet had been around for quite some time now. The problems that had been identified earlier in this magazine and by other industry insiders surfaced up with prophetic accuracy.

According to the reports, PTCL’s Pakistan Internet Exchange (established in August 2001) network – which, for various reasons described later, is the only Internet backbone in the country – was brought down to crawling speeds or complete halt for 63 times in the past two months of which 34 outages occurred in the month of April 2003 alone. The reliability of the PIE network nosedived from its typical 99.6% uptime to 93% which comes to be 50 hours of outage per month. These grand scale network failures have also severely damaged the country’s image as an IT-friendly country. A handful of Internet service providers with private circuits were able to partially restore their services and the rest that relied solely on PIE were out of service.

Each time the network was down – bringing down the entire country’s Internet access to a grinding halt – the outage was attributed to Denial of Service (DoS) attacks being executed from the Internet onto the PIE network.

There is no proof that each outage was actually caused by these DoS attacks as the industry has raised doubtful eyebrows and has suggested that PTCL has got a scapegoat in the form of DoS attacks which is being used to cover some other problems that have nothing to do with DoS attacks such as delayed troubleshooting response, poor implementation of Internet censorship and the mismanagement and goof-ups of the incompetent PIE staff. On April 4, for example, an optical fiber cable in the suburbs of Karachi was cut and resulted in the outage of bandwidth for the entire PIE network. It took more than 12 hours for PTCL to detect and fix the fault and during all this time, the ISPs were being told that a DoS attack is taking place!

Internet by definition is meant to be tolerant to physical disruption of individual network segments. There is no reason why this shouldn’t be the case with the Internet in Pakistan. To get to this point, we will have to ensure that instead of having a single point, we must have multiple entry and exit point for the national Internet access. Theoretically, while even today technical options exists that would enable ISPs to connect to some other (than PIE) bandwidth source, the tariffs for international private line circuits (IPLCs) have been crafted in such a way as to make them economically unfeasible. The industry has been long screaming to bring some rationalization to the IPLC rates without any success. Rooftop VAST connectivity (full duplex) for Internet bandwidth purposes is also a viable option to introduce diversity in connections. Rooftop satellite connectivity for Internet connectivity should be immediately allowed to the willing ISPs. This will automatically diversify the sources of Internet bandwidth in the country and the limited point of failures that the national network currently faces would be replaced by multiple entry and exit points. India has already allowed this and is reaping the benefits. The only proven way of remaining unaffected by DoS attacks it to have plenty of bandwidth available through multiple entry and exit points on the Internet.

Making good use of adversities, let us review the happenings and draw some conclusions in the light of the problems that we faced. For one, the repeated outages in the Internet access exposed our network vulnerabilities and design issues (technical and political both) like a frank teacher.

Let us start from a short primer of Denial of Service or DoS attack. DoS attacks fall under the broader category of hacking activities. DoS attacks are typically aimed at servers connected to the Internet with the intent of degrading or disabling the systems to the extent that the services become unavailable to legitimate users. Instead of attempting to hack into the target systems to access confidential data, DoS attacks focus on overwhelming the systems with bogus and/or defective traffic that undermines their ability to function normally.

While detailed classification of DoS attacks remains an interesting topic of a term paper, a quick-and-dirty one would divide these attacks into two types: resource-consuming and flaw-exploiting. The ‘resource-consuming’ attacks overwhelm scarce and non-renewable network resources such as bandwidth and operating system resources of the target computer. During the time when a DoS attack is in progress, legitimate traffic cannot get to the target server and its users are denied of the particular services that were being provided by the target server. TCP-SYN, Trinoo, TFN and TFN2K, Stacheldraht, Smurf, Trinity, and Code Red type virus attacks consume (typically limited) bandwidth of their target computers thereby rending them useless for the duration of the attack.

On the other end, flaw-exploiting DoS attacks like Tear Drop, Land, LaTierra and Oversized Packet attacks exploit known bugs in the network layers to cause the victim servers to behave abnormally, slow down or crash altogether and thereby denying the normal services to the Internet users. Flaw-exploiting DoS attacks are also called asymmetric DoS attack because of the fact that the launch pad of such an attack need not have equal or more network resources than the target machine. For example, the teardrop attack can be launched from a slow, dialup machine and can bring down a high-end server sitting on high capacity network connections.

For most of the part, the recent PIE outages were due to excessive traffic filling in the relatively small and finite PIE bandwidth of 155 mbps thereby denying the local Internet users to access the Internet as well as making their target websites unavailable. This excessive traffic was attributed to the release of some viruses on the Internet that are being considered the works of Indian hackers. These viruses generate excessive, bogus requests towards the various well-known websites and portals of the Government of Pakistan such as pak.gov.pk, infopak.gov.pk, islamabad.net, pcb.gov.pk, nab.gov.pk and nrb.gov.pk etc. As these viruses spread from one host on the Internet to the other, the target websites get flooded with bogus traffic intermittently. As hosts on the North American part of the Internet typically have bigger bandwidths, when they get affected with such viruses and start generating bogus traffic, they can quickly consume the finite bandwidths such as those acquired by PIE.

Government websites and portals have their majority hosted physically inside Pakistan. A large number of Government of Pakistan’s websites are hosted by a semi-government ISP (COMSATS) which hosts over 200 official websites and is a big bandwidth customer of Pakistan Internet Exchange (PIE). PIE thus carries these websites and portals on its back. As long as these websites remain hosted at COMSATS (and thus PIE) any attack targeted towards Pakistan Government’s websites would result in the congestion which would not only render these sites useless but would simultaneously result in the general Internet access being hampered for the common Internet users. This happened around 63 times during the past two months and the entire country suffered from Internet outages.

Keeping in view, the prevalent enthusiasm about the recent governments about IT and Internet, technical circles had already advised the government to set up a purpose-built facility to host government websites and portals. The suggested facility would centralize the hosting of all government portals. Manned by professional and specialized staff hired at market rates, the facility will have state-of-the-art network management hardware & software and its own high capacity network pipes connecting it to multiple exchange points in Internet over different carriers thereby ensuring high security and minimum outages.

Availability of such a facility would have multiple positive impacts such as better security for the government sites, ease of deployment, centralization of scarce IT resources within the government sector and network segregation of public access network and government networks. This suggestion, which appeared in this magazine earlier as well, still stands valid and is begging attention of the concerned Ministries.

Coming back, the existing set up of PIE is all too interesting. It lacks quality and professionalism in every aspect – from network design to hardware and from staff qualification to operating procedures. Had this set up been there for the internal consumption of PTCL, there could have been little objection – but not when you consider that fact that the entire country is going to get its Internet access from this arrangement.

For its PIE network, PTCL upstream service provider has been Singtel of Singapore from which an STM-1 (155 mbps) link has been procured. For backup (albeit highly unbalanced) an E3 (34 mbps) link connects PIE to BT/Concert. PIE network segments of Islamabad and Lahore are based on E3 (34 mbps) links that terminate on the same router that terminates the STM-1 link. An E2 link (8 mbps) from EMIX also terminates on the same router. On top of all this, circuits for the local ISP customers also connect to the same box. Hence, at PIE Karachi, the situation ended up with a single router bearing all the high speed upstream connection as well as the customer circuits. No specialized hardware is available to provide real-time insight into the network traffic flowing through the system which is the first thing you need to address any DoS attack or security and stability threat.

It is still unclear whether it was a paucity of funds and resources or simply bad designing that no thoughts were given to implement the PIE network on the more prevalent and proven model where the network is divided into core, distribution and access layers each having its own set of routing, switching and management devices. Had this designed been followed – which is now been reportedly given some thoughts at PTCL, the impacts of DoS attacks would have been limited to certain extents. For example, during a number of DoS attacks that targeted the PIE router itself, the inter-ISP traffic exchange also stopped. Had there been layers in the PIE network, while these DoS attacks would have resulted in Internet access going down by the local inter-ISP traffic could have kept running – one of the basic intent of having a local traffic exchange!

Till Feb 2003, PTCL had a local professional company of good technical repute looking after its maintenance of PIE under a year long contract. However, as the one year maintenance contract ended in March 2003, PTCL decided not to renew the contract and the day-to-day management of the PIE network fell into the unskillful hands of PTCL’s own employees. The recent Internet censorship orders by the Minister of IT&T were yes-sired by the PIE staff which used crude and highly ineffective means of implementing the censorship order. The measures taken for implementing the censorship are incredibly crude. Legitimate websites such as Open Directory (www.dmoz.org) found itself behind the locks just because the IP address of the site happened to fall in the same range of some pornographic website and the PIE staff decided not to take the pain of finding the exact IP addresses and just ban the entire IP address ranges! And this is not all. While the outages caused by DoS attack managed to get the public attention, glitches caused by inexperienced and untrained staff at ITI affect individual ISPs n almost daily basis and go unnoticed with only the suffering ISP feeling the pain.

In the words of the Minister of IT&T quoted in the online press, ‘PTCL should recognize its inexperience in handling these issues’ and that ‘its operations team comprises of non-professionals hired on the basis of sheer nepotism.’

Keeping these facts in mind, the industry has been logical in demanding a complete audit of the PIE network from a neutral third party. PIE management needs to grow more professional, more autonomous and less bureaucratic. It appears appropriate to suggest that the ITI division of PTCL should be spun off as a more professional and autonomous structure (similar to Ufone). ITI will thus remain part of PTCL as well as become more customer friendly and market competitive.

By the time of our going to the press, the Ministry of IT&T and PTCL have announced a number of welcome steps (that are still words, not actions). These include an advertisement for bidding of ITI Audit, plans of hiring foreign security experts to address the DoS attack problems, plans to revise IPLC tariffs, procurement of a second STM-1 (155 mbps) for the PIE, plans of making National Telecommunication Corporation (NTC) responsible for the hosting and bandwidth requirements of the Government websites and portals, allowing major ISPs to shift 50% of their PIE bandwidth from Singtel to FLAG Virtual Point of Presence, and intentions to host major government portals in US based Data Centers. Let us hope that things will finally settle in a better shape and let us thank the Indian Snakes for making all this good happen to us!