Steve Waddington's Network Notes

... is possibly the single most useful tool for any Cisco based network. And we have made extensive use of NetSense scripts for the last five years, using them to automate just about every repetitive task on our network.

This has included some complex operations such as load balancing of circuits, the application of time of day access lists, as well as mundane (but of life saving importance) tasks such as twice daily backups of running code on all our routers. The scripts themselves are straightforward shell scripts run from cron jobsthat call the various NetSense utilities. Here is an example from a set we used a few years ago:

# run the bandwidth control auto update script

25 16 * * * $HOME/net-sense/templates/control-scripts/auto-do-yellow-orange-remove-tasks.sh #1>/dev/null 2>/dev/null

# make the fairplay pool and load it

20 23 * * * $HOME/net-sense/templates/control-scripts/make-fairplay-cmd.sh #1>/dev/null 2>/dev/null

# add yellow, orange, red group traffic controls

03 08 * * * $HOME/net-sense/templates/control-scripts/add-rate-limit-tasks-yellow.sh #1>/dev/null 2>/dev/null

02 00 * * * $HOME/net-sense/templates/control-scripts/add-rate-limit-tasks-orange.sh #1>/dev/null 2>/dev/null

01 00 * * * $HOME/net-sense/templates/control-scripts/add-rate-limit-tasks-red.sh #1>/dev/null 2>/dev/null

And for backups, still in use:

14 19 * * * $HOME/scripts/bin/get-router-config.sh

The shell scripts (no reason to use bash, other than my limited programming ability) looked like this:

#!/bin/bash
#
#
cd ~/net-sense/templates
~/net-sense/bin/copy_to_tftp -rf all-routers -ipaddr 220.233.0.25 -pw logins.var -tftproot ~/tftpboot -subdir `date +%y%m%d`-1

where copy_to_tftp is the NetSense utility, all-routers is a text file with the ip address of each router, and logins.var matches the router IP to the login passwords. The router config is then copied down to a file tagged with the date stamp of the backup. So simple, and so useful.

The time of day traffic controls we used to use for different end user groups is even simpler:

#!/bin/bash
#

#
# add rate limit to CORE router interfaces
#
cd $HOME/net-sense/templates
$HOME/net-sense/bin/config_devices -log ~/logs/rate-limits_update.log -rf CORE-ROUTERS -pw logins.var -cf add-core-rate-limit-orange.cmds

Another script from the database would take the opt-in or excess usage information from user accounts and generate a list of IP addresses that the server would fetch each day, which in turn were used to create the .cmds file used by NetSense. The cmds file contain just straightforward IOS commands that the config_devides utility writes to each router in the CORE-ROUTERS list, and looks like this:

interface GigabitEthernet0/2.30
rate-limit output 1000000 10000 20000 conform-action transmit exceed-action drop

The rate to apply being based on the number of IP address or whatever other parameters were needed, and then generated automatically by the script.

While another takes the IP addresses and parses them into an access list so the content of 'add-core-rate-limit-orange.cmds' looks like this:

no ip access-list standard orange
ip access-list standard orange
permit host 220.233.4.x
permit host 220.233.11.x
permit host 220.233.25.x
permit host 220.233.9.x
permit host 220.233.14.x
permit host 220.233.13.x
permit host 220.233.4.x
permit host 220.233.9.x
permit host 220.233.19.x
.
.
.
!
deny any

The access list itself is not used within the traffic shaping. Rather, is is used to determine which path the traffic takes withing the core network. Which in turn is achieved by VLAN circuits and IGP announcements - the access list is used by the IGP (EIGRP in this case) which is given a metric lower than the default metric of the network.

This not only automated a time consuming task, but also allowed us to run very sophisticated time of day bandwidth controls on 7301 and 7206 class routers at full line speed traffic volumes (approaching 1gbps), while keeping the router CPU to sub 60%.

The upgraded cache was installed just before Christmas, and production traffic progressively added over New Year and January.

Our biggest problem so far has been to reduce the amount of asymmetric traffic traversing the cache switches.

In a nutshell, the PeerApp system takes IP traffic from all our supplier circuits, decides, via port number, which traffic is cache-able, and redirects that traffic to the cache servers. The cache servers then perform further analysis of the traffic, storing what can be stored onto disk, or delivering what can be delivered off of disk, and passing through the rest of the traffic back to the network.

Very simple in terms of any 're-engineering' we have to do to accommodate the cache - which is to say, virtually none - just plug the external circuits into the cache switch rather than the border router, and then take the same number of circuits out of the cache switch and connect them to the interfaces the external circuits occupied previously.

However, our traffic is not as simple as that. Because, as we have built the network out, we have done so with a blend of centrally supplied (from Sydney) and locally supplied (in each state) bandwidth, based, at each point in time, on the most cost effective delivery method.

The net effect was approximately 20% of traffic through the Sydney border router was asymmetric. That is; inbound packets arrived on a different router to the outbound packets.

While this causes no practical problem for user Internet access, for the PeerApp cache it reduced the effectiveness of, particularly, port 80 caching - which is the main area of improvement we looked for with the new cache system.

Heaven forbid the life of a network engineer should ever become boring. And over the last four weeks our core network team has progressively removed all but a tiny remaining fraction of asymmetric traffic from the NSW network.

End result?



The new cache is now generating twice the traffic of the system it replaced. With 2.2Gbps of traffic offered, we see peak generation of 850Mbps, and an average of just under 400Mbps generated traffic.

The graph peak coincides with the start of our 'off peak' or uncharged download period. The fast decay of the traffic is, I believe, due to the quick delivery of cached content that is requested. Which in turn matches the 'user experience' of near wire speed (6Mbps in my case) of data delivered directly from the cache.

... will lead to an ultimate loss for everyone else, especially the ISP industry.

Why do I think that? Because what Justice Cowdroy's judgment says to me, and from the look of this, the rest of the world as well, is that the Australian ISP industry has proved itself unable to take reasonable steps to behave like a responsible corporate citizen, and, if the industry wont be responsible for itself, then a change to the legislation is needed to force it to do so.

I thought Cowdrow's admonishment of the iinet testimony during the trial was a clear indication, and warning, that 'playing dumb' was not a good idea, and that some future competent prosecution could have a field day with.

So here it is; on the Lateline story last night Neil Gane said "We are confident that government will not intend a policy where rampant copyright infringement is allowed to continue unaddressed and unabated across the internet.". Which to me indicates the court case was only the first battle for AFACT in the war of copyright infringement. Having proved ISP's wont take any responsibility themselves, and the current law (subject to whatever appeals may or may not happen) can't force them to, they will lobby for the legislation to be changed.

Does no one play chess any more or understand what a gambit is?

I guess whatever will happen will happen. A pity though, it would have been so easy with a little less ego and taking a little more responsibility to have arrived at a far better outcome for everyone.

.. could it be Sri Lanka?

No kidding. Deep in the jungle in the Bay of Bengal, I was checking email, accessing the company intranet and had a measured download speed of 1.6Mbps.

I can't say Sri Lanka is the best for mobile broadband coverage, since I have never been to Madagascar, Myanmar or the Congo, but I have been to Pemberton, Wagin and Gnowangerup, and the coverage in Sri Lanka is better than there.



Apart from coverage what is the difference between 3G in a 'third world' country and Australia? Well, in Sri Lanka, the much lower per capita income means the carriers can only charge $5 per month (500 rupees, which is close enough to A$5 at the current exchange rate) for the service.

A very interesting number, I thought, because if you think about it, it has to be an honest reflection of the true lowest cost such a service can be offered to the retail market, without wanker bundling and/or gouging download pricing.



Ah, but, you say, Sri Lanka has much higher population density and therefore a much better economy of scale! A much touted argument, so much so that like any cliche who ever questions it?

So let me tell you, six hours drive to get to a place is still six hours drive - whether the distance traveled is 600km or 200km - the effective cost is still the same. Sure Sri Lanka has a much higher population density, but higher per subscriber per square km than, say, New South Wales or Victoria? It would be hard to see how.

Less competition? I don't think so. There are three competing networks in Sri Lanka too. Lower cost of bandwidth? Doubt it. I don't know what a good wholesale price for bandwidth is in Sri Lanka, and maybe we don't buy very well there. But I do know that the price we pay per Mbps, taking the best quote from a number of suppliers, is around the same as the price in Australia in 1999.

Maybe its just me, but if a government was determined to spend $43bn on national broadband infrastructure, wouldn't it be preferable to have a goal of making access cheaper and more widespread, rather than more expensive and only for cities and large country towns?



Mr Walker, perhaps you need to relocate your skull cave to where the 21st Century pirates are.

Last week I was in Sri Lanka, part of the trip agenda to see the site of the Model Dairy Farm Project. The site, in the so called 'dry area' Matale district (still looked like lush rain forest to my WA wheatbelt adjusted eyes), is six hours drive - 200km - from Colombo.



On the advice of our manager in Sri Lanka, we broke the trip into two parts. Leaving the Colombo Hilton at 1pm on Tuesday we headed for Kandalama, planning to arrive at about 5pm. From there we overnighted at the Heritage Hotel and were picked up at 7am by a SLWCS four wheel drive for the remaining two hour drive to the site.



As we were driving we got to see first hand the 'conflict' areas between farmers and elephants, and had explained to us the traditional methods farmers use to cope with the problem. Namely, building a tree house on the edge of the jungle, and staying up all night to watch for elephants. Should an elephant, or more likely a herd of elephants come along, the idea is the farmer in the tree house makes a lot of noise to scare them off. Which doesn't always work. In which case the farmer can either choose to lose his whole crop, and income, or climb down the tree and beat the elephants off with a stick.



I can't imagine the courage it would take to set to a group of wild elephants like that. I suppose if the alternative is total ruin, people can do some pretty amazing things. Even more amazing though, is that often it works, and the farmer can successfully defend his crop. Tragically, sometimes it doesn't work, and the farmer is killed or injured (50 people a year are killed during raids), and/or the elephants destroy the crop. At that point, things can turn nasty for the elephants, as the farmers family may start a vendetta and hunt and kill the elephants (estimated around 100 per year).

Humans and elephants have existed side by side in Sri Lanka for centuries, and farm raids were only very infrequent. What has changed in recent years, and made raids far more frequent, is three things;

1) Encroachment on native habitat by logging

2) The spread of a type of non-native grass that has far lower nutritional value than the native varieties it has replaced

Which both contribute to less food for elephants, and in some cases starvation, and

3) Farms planting cash crops of pineapple, corn and legumes - food elephants, particularly starving elephants, love.

Now, there are an estimated 4,500 wild elephants in Sri Lanka, but each year, the population declines by several hundred.

The SLWCS have initiated a number of innovative projects and trials, all with the same hallmark of improving or changing 'agricultural practices so that agricultural development can be sustained over the long-term'.

Hence we have the project we have become involved in - the Exetel Dairy Project - which, in a nutshell aims to:

* cross breed native and Indian cattle varieties to improve milk production from 4 litres per day to 10 - 15 litres per day. At this level, dairy farming becomes viable and can replace cash crops for farm income.

* Elephants and cattle co-exist, one eating the fodder the other leaves

* Sri Lanka imports 90% of its milk, improving local production will help the economy as well

* develop farming practices that minimize the impact to the local flora an fauna

* based on the outcome of the trial project, provide education to other farmers of sustainable dairy farming

* Increase productivity, which is decreased when the farmer has to guard crops all night

We arrived at the site by 9am and were greeted by typical Sri Lankan hospitality - a table set in the shade of a mud brick hut laden with local treats, and rich, strong Ceylon tea - thoroughly delicious and greatly appreciated.



It being day 20 of month one of the two year project, there was not a lot yet to see at the site. Never the less, preparation work had commenced, and we walked the length and breadth of the boundaries talking with Chandeep Corea (Operations Director) and Samantha Mirandu (Project Manager) who explained the project in more detail.



What was abundantly clear was the enthusiasm and dedication that the project team has towards meeting the project goals. It is one thing to be told (albeit from a very trusted source) that this is probably the best way we can contribute to conservation in Sri Lanka, it is quite another to see first hand that there is no doubt it is.



A couple of points of interest; as we were walking along the track at the site, Samantha pointed to a large round indentation in the ground, 'Elephant tracks' he said. 'How long ago do you think?' I asked. 'About half an hour'. We didn't see one, but it was pretty clear from those and other tracks, that, at any minute a wild elephant could emerge from the jungle. What do you do if that happens we asked? Run very fast, apparently.



What we did see though, was wild pea fowl, a peacock with attending pea hens. Fascinating to see those birds in their natural state in the wild. Also, just briefly along the side of the road as we were driving was what looked like a rooster, but with a long, almost peacock like tail, and four, what looked like, pea hens - native jungle fowl as it turned out.

Didn't work out as we had planned. Everything was full prepared for the cut over at 3am this morning, but, the best laid schemes etc.

What went wrong (Wha' gang oft' agley? It is nearly the time whatever is misquoted from Robbie Burns' original words will be more than made up for by inebriation, after all) in this case was two things:

1. Of the four GBIC interfaces needed to carry traffic between the cache and the border routers, one, which worked fine in testing yesterday, wouldn't establish a connection. Nor would the spare. So we had to figure it was a port problem with the port itself.

2. All four ports on the cross-connect switch reported zero bytes per second throughput, even though the interface counters were incrementing bytes in and bytes out. Not a fatal problem in itslef, but still, an indication of perhaps some other problem or even IOS bug.

So those two issue ate up the maintenance window time before they could be resolved and that was that for the morning.

Next step is to retest and fix whatever is causing (1), and fully understand/fix the reason for (2). Which means the cache wont be commissioned this week, and most likely not until at least next Thursday. Although, it may even be better to wait until the period between Christmas and new year, which is the lowest usage period of the year.

We will have our weekly network and bandwidth review meeting tomorrow and decide then.

Most of the new server equipment arrived on Friday, the remainder coming today, along with the PeerApp engineer who will be working with us on the installation.

All of the equipment is now at the POP, with the estimate being that it will take another full day to complete the physical installation, and remove the old equipment. If that work out, then the maintenance window fvrom 3am to 6am Wednesday morning is our target timeframe to commence sending traffic requests to the new cache.

On complication that may delay things however, is that we have a pending 'statutory carrier obligation' request which has to take priority over other work. If that work turns out to be more complicated than we expect, then we will have no choice but to see it through, by necessity holding up the cache setup completion.

Worst case, I would say that we should be able to see early results by the end of the week - or perhaps over the weekend, depending on how long it takes the cache to populate.

.. well the ones that matter anyway. Which I take to be the oft quoted '90%' of Australian households.

While Conroy is going on about the brave new world of an NBN, and how if the Luddite detractors had their way, the 'government' would not have built an electricity grid. (It was J.P Morgan, Westinghouse and General Electric in the US that paved the way, but lets just suspend historical fact for the sake of a political sound bite). Why is he being so defensive if it is such a self evidently good thing?

The spin has clearly changed from 'Everybody wants this' to 'it is up to the government to do this for the good of all (even thought they may not think it is such a good idea now)'.

Let's put aside that 100Mbps+ bandwidth is already being delivered to many households, and many more could have it if they wanted. That upgrading existing infrastructure to accommodate bi-direction, true IP, traffic can be done with the upgrade of a few boxes (ok, a few thousand boxes, but for well less than $1bn in any event).

The thought occurred to me, why settle for 100Mbps? Whether it is fibre to the node, or fibre to the house, where does the magical figure of 100Mbps come from? 1Gbps speeds have been common, and cheap, for half a decade now, on both copper and fibre.

So what would it cost to deliver 1Gbps to every (90%) house in Australia? I did a few 'back of the napkin' sums, and, using current retail (or street) pricing for equipment, my reckoning comes to a shade over eight billion dollars.

That is for fibre to a media converter to each house (8,000,000), terminating at each exchange (1,100) on a 1Gbps switch port, aggregated into a suitably large router for backhaul either to a central point in each state, or to the call collection area (66) for the region and then to a central location for the state.

To calculate the termination cost at each house, I used the current price Telstra would charge for a new service, with a mark up to allow for the extra termination cost of fibre.

Balanced against all the costs I have trivially over looked, has to be the fact that I based all the capital equipment costs on the retail price. One would have to assume some sort of discount would be available for an order of eight million. Also, that a contractor terminating every house on every street is going to charge less than the once off special truck roll the Telstra pricing for installation reflects.

But, $8bn is still a lot.

Actually, no it isn't. On a 5 year ROI, it is $17.03 per month per household.

1Gbps to my house for around $20 per month. I think I can see the value in that.

Who has the failure of vision now Mr Conroy?

(apologies to Bob Dylan)

This year, more than any other, I have noticed a change become apparent. And by a 'change' I am a fundamental shift in the paradigm of business in the service provider industry.

I am never quick to pick these things up, so I can only assume that if I notice a change, it has well and truly already happened, and many other people are enacting/changing/making their plans accordingly. Maybe they are all keeping quiet about it, or maybe they are in denial, but from what I see, there is little evidence of accommodation for the change actually being made. Maybe it is something that will be more noticeable after Christmas.

What is this mysterious 'change' then?

Perhaps it is best explained by it ultimate effect - which is not yet, and there is still quite a while, before it is fully realized. So here is ti:

"There will be no job in Australia for anyone who is not directly contributing more to the GP of their company than their employment cost"

But surely that is already the case? No efficient company employs anyone if there is not a commercial reason to do so - at least in theory. Which I am sure is correct, if not always in fact, then at least in desire of the people controlling the finances.

However, how many people go to work each morning thinking 'Today, the company I work for is going to pay me 'x' hundred dollars, so what will I do today to ensure the money is there to pay me'?

I would take a bet that a far more common attitude is 'Work today, only 'x' days to go until the weekend', followed by whatever minimal effort required to do what they are instructed to do, while attending to the important things like Facebook and MSN.

Which has been the way for decades, at least, of employees doing the minimum required on the expectation that the 'big rich fat cat company' they work for will keep paying them (them or someone else, but better the devil you know), and employers accepting that.

I am not arguing that is it right or wrong, or that is is true in every situation. I am saying that it is my observation, from what I have seen, people I have spoken to and heard, over many years.

I put forward the view that this attitude/acceptance is, if not the, then a major driver for for many companies outsourcing to countries where wages are much lower than Australian award wages. If as an employer, you have resigned yourself to pay staff to do a minimal job, would you rather pay them $1,000 per month or $5,000 per month? A no brainer, which is just as well, given the very low amount of brains that led to that situation in the first place.

For a brief time I worked as a consultant for a 'big five' IT consulting firm. It was very clear, as a senior manager, just what was needed to stay employed there. Time accounted for in six minute blocks, 80% of time billed, and get as many of you colleagues as much billable time on any job as well. Every business run as a practice knows those figures - without billable hours paying for staff, you just can't have the staff.

Why is it so much less 'cut an dried' for a service provider? Is it really such a hard exercise in abstract thought to see that the time a network engineer/sysadmin/programmer/support person spends doing things can't relate directly to the bottom line of the business. And if that major hurdle of thinking can be overcome, then surely it is a small step to a) relate it to real contributing value to the business and b) work out how to ensure you are valuable enough to overcome the competition (being the people who will work, very happily, for 1/5th of your salary).

What I can say is that I can define a single point as the big change in my career from being 'just another engineer' to someone who highly valued, given the most important projects and promoted, by whatever company I worked for. And that was, one day (I am not sure of what exact time period the thought crystallized, it was over 15 years ago) I embraced this thought "Whatever the company I work for pays me, I will try to ensure I return 10 times that in dollar value". Which is a very simple way of putting it, so I will qualify it a bit:

- if I was on, say $50k pa salary as a pre-sales engineer, I would ensure the sales I contributed to exceeded $500k pa
- If I was responsible for CAPEX for a project, I would ensure the ROI was as short as possible, and look for enough projects over a year to return more than ten times my salary
- when first starting a new job, I would look for an immediate cost reduction (which I never had any trouble finding) that would at least cover the cost of my starting salary

As a junior engineer, there was little opportunity to have much control over what work/projects etc my employer of the day could offer. But the funny thing is, once I had embraced that mode of thinking, from that point on, I did not stay a junior engineer for very long.

At first look, two problems I see are:

* how will it handle captive portal systems?

These are an increasing part of ISP operations, redirecting end user traffic for any number of reasons. End users switching to the Google public DNS servers will cause portal redirects to break, most likely resulting in a hung browser session or 'page not found' error

* how will it handle internal IP address maps for private networks?

Some ISP's, and many corporate networks make extensive use of RFC 1918 IP address space. Company intranets are a prime example. Mapping that through a local DNS resolver is no problem. Completely breaks with a public resolver though.

It is interesting to see a selling point from Google is that is will "Get the results you expect with absolutely no redirection.". I assume, one day, Google plan to make some revenue from this system. I wonder how they plan to do that?

I was driving with my oldest child yesterday, who has just completed his first year of Uni. As a teenager, young adult, FPS and MMO gamer and student, it is hard to imagine any demographic that a) would make more use of the Internet in their day to day lives, and b) would have a better lay persons understanding of the Internet and broadband.

So he was telling me that the consensus amongst his friends was that the much high speeds provided by fibre would mean lower game pings. Taking a typical Call of Duty in game ping time from 80ms down to 20-30ms.

I found that comment very interesting for a couple of reasons.

First, I asked him 'What about download speed?'

To which I was pleasantly surprised to hear that download speed depends more on the speed of the server being downloaded from, 'and anyway, we have 12Mbps don't we, so stuff downloads about as fast as it can?'

Further questioning revealed the first part of his comment came, he thought, from something I had told him at some time (so kids do listen to some things we tell them, wow). The second part was from comparisons he and his friends had made based on their various broadband connections and service providers, and that a uni tutor had told him the University network limited external connections to 4Mbps per user.

Download speed of current copper delivered broadband doesn't seem to be an issue - at least for one particular group of, semi well informed, users.

Next I tackled 'how will a new NBN improve your ping times then?'

'Well' I was told 'In the US everyone has T1 or T2 connections and their ping times are 3-15 ms.'

Uh huh.

In this context, 'T1 or T2' was meant to mean 'fast fibre connections to the home', so for the sake of maintaining focus on the core issue, I let it go at that.

I went on to explain that ping time is a function of user distance from the server, and once the line speed went over about 256kbps, faster speed made very little difference to the ping time (which _we_ all know, of course). I likened it to a freeway (since we were driving down one at the time), whether it was two lanes or four lanes or twenty lanes, the speed limit was still 100kmph.

'But can't the data go faster than the speed limit if there is more room?'

'No son, the laws of physics police that one' I informed him in sagely father tones.

It required a little more explanation than that, but the concept was quickly enough understood.

Never the less, the question remained that wouldn't it still be better to have 100Mbps Internet than 12Mbps?

'Possibly' I said ' but what would you use it for?'

Receiving the stock answer of 'movies, TV and streaming multi media'.

So I went on to explain that we already had a connection of well over 300Mbps bandwidth when we had Foxtel. And now with free to air digital TV we still have well over 100Mbps of bandwidth for the express purpose of TV and movies. Foxtel IQ and Tivo filling in the 'time shift' gap, if there was one, left by the demise of the VCR.

I am not sure I fully got the concept across that broadcast media is just the same as Internet bandwidth when there is no, or minimal, need for real time interaction from the user. However, certainly for the time we were driving, we couldn't come up with a valid example where it would not be equivalent (having excused P2P downloads as something to be addressed by the content providers in future - that is, if the content is readily available to meet consumer demand, P2P of pirated TV and movies will become irrelevant).

The conclusion we arrive at therefore was the _desire_ for faster broadband for the informed residential user was literally, to make the ping times faster rather than to have access to more bandwidth. But, as we see, that is a delusional wish based on misunderstanding the physics. Assuming $43bn could ever be justified for a 20ms ping reduction in Call of Duty 4, of course.

After quite a bit of discussion, checking and double checking, we have reached and agreement with PeerApp to go ahead with the upgraded platform in Sydney and also add new equipment for Melbourne and Brisbane (since our traffic is now at levels in those states that make caching worthwhile).

The equipment will be the latest iteration of the Ultraband product we have been using, which upgrades:

- bandwidth handling capacity in Sydney
- the ability to better recognize cachable protocols, and improvement to the traffic redirection to the cache engine
- allowing the inclusion of multimedia and other web content

We are expecting the equipment for Sydney to arrive and be installed mid December, with the goal to have the equipment in trial phase before Christmas. Melbourne and Brisbane installations will subsequently take place later in January.

With the upgrades to the equipment, plus increased disk capacity, we are expecting cache generated traffic to be in the order of 600Mbps in NSW, 250Mbps in Victoria and 150Mbps in Queensland.

For organizations with many people in the filed using HSPA, we can assign all IP addresses to those services from a single IP address block.



Alternatively, we can assign each service from a private IP address block. This may be an advantage where the client wants to strictly control Internet access for all users or restrict access via a central firewall only.

Has anyone thought about who or what a 'hacker' actually is? Apart from the root of all evil (in IT), a justification for a huge security budget, and like every bogey man before them, a way to scare children allow hacked phrases to pass for policy by those too lazy to spare a moments thought on the subject.

I saw this report, and it just made me wonder; instead of reporting "Power outage caused by hackers", would it not me more accurate to say "Power outage caused by IT idiot unable to secure vital infrastructure systems from teenagers"?

No, no - far better to blame the outage on these mysterious and vapourific 'hackers'. Because everyone knows what a hacker is, how inherently evil they are, and a two day outage is just the sort of mitschiefe they would cause. For no other reason than they can. Whoever 'they' actualy are.

It is very convenient that invoking the word hacker seems to completely negate the need for any further investigation into the cause. Everyone knows you can't do anything about hackers, right? And I am pretty sure by now the logs will show whatever they need to show to support the 'hacker' case.

So lets just say it was these scallywag hackers for a moment. Who are they meant to be exactly?

No one extorting money, or doing it for any other reason than vandalism it seems.

That would seem to put it in the realm of teenagers with a pretty good Internet connection and too much time on their hands. And we can all agree there must be lots of people like that.

Exclude the ones who idle away their time with WoW/Quake/Halo etc. Which must really narrow down the field a lot. Exclude those that are not inherent vandals (my kids and yours, at least). Then exclude those that are vandals, but express it with a spray can or a knife on a train seat.

What does that leave us with? Someone who wants to create widespread vandalism by, specifically, hacking into a public utility, with (the real kicker) the ABILITY TO DO SO.

Because of course, it is pretty easy to hack into a computer system, right?

Wrong.

It may be easy enough to hack into a pirated, unpatched, ten year old Windows OS running on a computer in China or Romania. I think however that the same is not as true for fully maintained servers run under the supervision of a competent IT person. Even without a zillion dollars on firewalls and other security enhancements, just the OS itself - Windows, Unix, Linux or whatever, is enough when run by someone reasonably competent.

But even forget about the inherent security in OS's. Look at it another way; saying a problem is caused by 'hackers' is the same as saying that a teenage vandal knows more, can out think, and can beat the experience of a (presumably) degree qualified IT professional who is paid by their company to secure against just such an event.

Well, any number of US TV shows and Hollywood movies show us just how common teenage geniuses are who can out think any corporate 'suit'.

Or, perhaps the true headline should read "Power outage caused by ass covering IT idiot, tries to blame it on hackers".

I think I fill out more than my share, because I don't mind doing it - on the assumption that whatever company has gone to the trouble of asking for it will probably, at least in some way, act to improve their service in some way.

Over the years, the most consistent, to my observation most acted on, and in my opinion, best put together, are the feedback forms requested by Cisco at the closure of every TAC case. In the past I have had direct calls from senior managers in the Australia and the US in response to - not even particularly - negative evaluations I have given. Leaving no doubt there was a genuine desire in the company to fix whatever it was that lead to my dissatisfaction in that case.

And, as many people would know, they really hassle you to fill out the form too. The last time I was in that loop it was three emails, of increasing urgency, along the lines of 'We haven't received your feedback yet, please do it now'

I think Cisco used to say that any customer response of '1' (the lowest response on the 1-5 scale) would be reviewed personalty by John Chambers. I don't know if that is still the case now, but nothing led me to think it didn't actually happen that way. Though smaller then than they are today, it was still an amazing thing to see in, what was then, a $15bn pa revenue company.

I don't know if Cisco set a new 'industry standard' for customer feedback in that regard, but certainly many companies in many different industries now have a similar approach. From just about every hotel I have stayed at in the last two years, to airlines, and even the dealership where I get my car serviced. Similar sorts of questions, similar sort of feedback and follow-up.

But here is the kicker - they ALL make one fundamental error (only my opinion of course) in the way their survey asks for information - Where the ask, and most of them do, a question about 'your expectation'. it will usually look something like this:

On a scale of 1-5, one being failed dismaly and five being far exceeded, how did our service X meet your expectation?

And there will be a bunch of questions about different areas of the product or service where the respondee is expected to give a rating on their 'expectation'.

One can only assume that somewhere along the line, those customer feedback forms contribute to or form a basis for and employee/manager/department evaluation. And fair enough too.

But nowhere I have seen, on any form, have I been asked to actually say, from the start, what my expectation was.

So what am I actually being asked to say? My assumption may be quite wrong, but the feeling I get (and the first hand information I was told by some TAC engineers many years ago), is that the goal or KPI is to get across the board '5's' for every response. Which means the the company has to 'far exceed' my expectations.

No problem, always nice to know that people have an incentive to exceed expectations.

But what happens if my 'expectation' was already for excellent service? And I received excellent service. And therefore to answer the question as it is asked, I really need to put a '3' - the service met my expectation.

It seems like a very big error in the data collection method to me. Maybe I have missed something fundamental in the way feedback is intended to be analyzed or some other subtly that completely escapes me. It wouldn't be the first time. Or maybe, the question/form structure is just being used by rote by company after company, with no real understanding of the true intent of the person who was the original source of such things.

An example of this is where last month I stayed at Star City while I was in Sydney. A really nice, top class, 5 star hotel (A real bargain at the off-season room rate I got on-line). My expectation was that the service would be great, the room would be first rate and I would enjoy staying there. And it was no surprise to find my expectation exactly met. Which is what I wrote on the evaluation form.

The fairly prompt email feedback from the hotel was 'We are sorry to hear you did not enjoy your stay'. But I did! My expectation was met in each case - the middle response on each question they asked. What I think they, and every customer feedback for like it, should do, is at the very start, ask what my expectation actually is.

True, it will mean the analysis of the response will be more complicated that just averaging the gross total. But surely that is what is needed if the intent of such feedback is to be fulfilled.

Anyway, the result of my musings on that subject are that I have redesigned the corporate support feedback page to this:

Please select a response from 1-5 for each question using the following guide:

1 - strongly disagree or far bellow my expectation
2 - somewhat disagree or bellow my expectation
3 - no opinion either way or just met my expectation
4 - somewhat agree or exceeded my expectation
5 - strongly agree or far exceeded my expectation

If you know the ticket number, please enter it here:

If you know the name of the person who handled your fault, please enter it here:

1. Before I logged the fault, I viewed Exetel as a high quality service provider

2. My fault was handled promptly and efficiently

3. The fault was fixed in the timeframe I expected

4. The fault has been resolved to my complete satisfaction

5. I was kept fully informed of progress at every stage

6. I think Exetel did everything required to ensure the fastest resolution

7. The way this fault was handled has raised my opinion of Exetel

And lastly, the other minor frustration I have with such feedback forms is, there is invariably no way to say 'you haven't asked me the right questions!'.

Hence; todays blog post.