Steve Waddington's Network Notes

This is the 3rd iteration of the high level diagram we will use in 2012.



There will no doubt be a few more tweaks before it sees commercial release, but I think it captures the essence of what we want to convey.

'Underneath' this diagram will be more detailed, but still conceptual, location and service specific diagrams, about 10 in all, that will show how services are supplied. And finally, there will be the engineering design diagram, individual to each customer, that is attached to a proposal, showing in detail what is being supplied.

Here is an interesting site.



Should give a few people pause for thought.

I have been thinking about how to redraw the Exetel network for sales and marketing material. The network is long past the stage where it can be represented in full, even taking treating each POP as just a circle and only putting in the WAN links.

The challenge is to show 'at a glance' the main features of a network with multi, multi GE and 10GE interconnects, dozens of peers, data centre locations and services, that spans three countries. At the same time contain no false statements (like using a port speed or supplier network capacity to represent a circuit speed), and be accurate; as a summary of a network not a 'lossy' compression.

Yet even if a diagram did that, it's not the whole story of the Exetel network. There is one massively important part of any network that rarely, if ever, makes it onto the diagram.

After a lot of thought, this is what I came up with:



In this iteration I haven't put in the specific WAN circuit sizes, for two reasons; One, given our no contention policy, the size of any one circuit doesn't matter, as long as it exceeds the peak bandwidth at all times. Two, Due also to the no contention policy, circuits are increased often - faster than the print/distribution cycle of a brochure. And, in any event, anyone interested in the fine detail of a location or data path can ask, and be given the up-to-the-minute value at that point.

Only a draft at the moment. It will be interesting to see how the bright and capable minds in sales improve it by the time it gets to 'brochure' stage.

No one can deny that walking around on a big flat field during a lightning storm is a dangerous thing to do. Quite rightly, workplace safety should be observed.

No one can deny that an aeroplane is made of conductive metal, and much, much taller than a person.

So all those planes sitting on the Perth tarmac yesterday were being constantly hit by lightning? It must have been terrifying for the passengers sitting on the planes.

The Perth terminal, last time I looked, has plenty of steel and metal superstructure, and is taller again than the largest plane that docks there. And I would be very surprised if it didn't have substantial lightning rods higher than that as well. It must have been deafening as bolt after bolt hit the building.

With so much metal sticking up in the air, the safest place to be would surely be on the tarmac apron between planes on one side and the terminal on the other, being effectively surrounded by a big Faraday cage.

But physics is clearly only one factor when it comes to workplace safety. Because on the other hand, sitting inside playing euchre all day because it was a bit wet out is probably a better choice.

Over a decade ago I worked for a wholesale IP supplier. It was only a small company, but, because of some innovation and foresight by one of the founders, ended up supplying about 30% of the non-Telstra bandwidth used in Australia at the time.

The great innovation that they pioneered was the use of uni-directional satellite transmission as an alternative to the, then, much higher cost of terrestrial bandwidth. As I recall, satellite bandwidth (on the old, wobbly PAS2) worked out around $5,000 per Mbps per month, compared to terrestrial (pre SCCS) at around $17,000 per Mbps per month.

With a delivery cost less than one third of the alternative, many ISP's found they could overlook the additional 200ms latency, the bi-annual sun transit outage, and the more frequent downlink signal loss (for any number of reasons). So despite the less than perfect service availability, the company had many loyal customers.

Until there was some competition, in the form of much lower cost terrestrial bandwidth. And the 'loyalty' evaporated in a direct relationship to pricing as the cable price approached parity with satellite.

I am not drawing any negative conclusions about ISP's making the only decision possible for them - which is to take the lower cost, better performing, more reliable product. It would have been stupid for them not to. Rather, it's just the fact that many people would start of justifying their decision by saying 'I have been your loyal customer for many years, but...'

Which made me wonder, how is that loyalty?

Loyalty would be where the customer just kept paying whatever was being charged, in the face of all good business sense, forever. Or a company 'loyal to its customers' forgives any non payment for services. Or a patriot, loyal to their country/ideal/football team, lays down their life for it, or at least continues to support it when they are at the bottom of the ladder.

That is loyalty, and that is its price.

So what is the real, commercial price of 'loyalty'? In my experience it can be defined as exactly one cent.

For example, many people I knew in my home town of Albany would go 'loyally' to the petrol station owned by their mate/relative. Until, the price was one cent a litre cheaper somewhere else. Then they would have no qualms about driving to the other side of town (only a 3 minute drive, but still) to fill up.

Or, why are prices $x.99 and not $(x+1).00?

Going back to the wholesale bandwidth provider I was talking about. Customers would remain loyal until the price per Mbyte was 1c cheaper somewhere else.

Still, I suppose 'loyal' is less than a mouthful to say than 'I have been your opportunistic customer, while your price was the cheapest and while it suited me'. Not that there is anything wrong with that. That's what we all, as customers, do. Maybe it is even hardwired into our brains - to constantly weight the price/quality/value, and feel ill at ease if we do not have what we consider the optimal mix.

Interesting that of all the ISP's around in 1995, and still in business now, the most successful by far is the one that (as told to me by the CEO at the time) aimed to always be the lowest cost producer.

I was intrigued by this article, which seems to be advocating the benefit op IPv6 is to allow direct exposure of user (student) PC's to the Internet. An interesting view. I wonder if they have thought about that at all.

I can't recall the source, it was a few years ago now, but it was a reputable authority, and I had no reason to doubt it was true; that a new windows PC connected directly to the Internet, would be compromised and incorporated into a bot-net in less that the 20 minutes it would take to download the security patches from Microsoft.

My own view is that, without NAT, the burden placed on ISP customer support just to say 'Not our problem, you are responsible for the security of your computer' to customers that had been hacked by bot-net collectors would more than double the cost of Internet services. Assuming an ISP could even get away with saying that, which they couldn't, and some officious govt watchdog would no doubt see an opportunity to justify their meaningless existence and step in and make it mandatory for the ISP to fix that too.

Effectively, without NAT, ubiquitous residential Internet would not exist.

IPv6 solves the depletion problem of IPv4. And that's it.

Still, a great opportunity to make lost of consulting money either way. It looks to me like the Y2K of this decade.

[Aside: remember the panicked articles earlier in the year about 'The last IPv4 address will soon be allocated'. While they didn't actually say the Internet would end on that day, none of them went out of their way to point out that it wouldn't either. Well, it hasn't, has it. And shows no sign of doing so. But who gets column inches or sells copy saying 'IPv4 nearly gone, but not really a problem for a few years yet. Experts say'.]

Speaking of civic virtue and social responsibility, I was in Tokyo a couple of weeks ago attending NTT's annual conference. One of the presentations was talking about bandwidth use - predicting the, now unsurprising, exponential growth. But what was surprising was a completely unexpected (by me) sharp downturn in one area of bandwidth use, namely; P2P.

The graph shown was a measurement of OCN bandwidth since 2004, showing a consistent average of a bit over 40% growth per year for 'down' traffic, and the same growth for 'up' traffic, until 2009 where traffic dropped from 200Gbps to 140Gbps over the next two years.

The major contributor to the increase in down traffic was explained as the increasing use of streaming content. But why the puzzling decrease, against all trends, of up traffic?

Quite frankly, I found the explanation astounding.

Apparently, in 2009, the Japanese government issues a 'stern warning' to the nation about the security issues of P2P, and that is was not socially acceptable to download unauthorized copyright content. And so, people stopped doing that.

Draw your own conclusion about how strong the moral fibre of a society that can do that must be. My thought at the time was 'if only'.

Here is an interesting article from Akamai.

If average peak speeds are, as reported, 14.4Mbps, which takes into account people still on 1.5Mbps and lower ADSL circuits, as well as people on ADSL1+ and ADSL2 circuits, then by my rough calculation, it puts the average ADSL2 connected user on a peak download speed of around 16Mbps.

Compare that to the connection speeds published by NBN, with the entry level of 12Mbps.

What is the government spending all that money for again?

But what was the 16Mbps peak in 2009? Well, it's a long time ago now, but my guess is the take up of ADSL2 - the carrier ADSL2 network backhauls had less contention.

Any service provider using the NBN will be able to exercise control over peak use by the amount of bandwidth they order to connect to an NBN POI. The 'level playing field' price for that bandwidth, represents between a 50% and 70%, or more, discount on current carrier interconnect pricing (obviously depending on the carrier, and the company buying it).

Which would have to mean that any ISP now following a policy of zero contention could be safely assumed to have no real problem continuing that policy based on the NBN price model. And I can only agree with our Sales Director Glenn; broadband users are unlikely to increase their spend on service 'just because' it's the NBN. Rather, they are going replace an ADSL service with an NBN service, IFF the change over cost barrier is low. In other words, someone spending $35 or $45 per month now will take an NBN service at that price, but someone spending $35 today is not going to suddenly say 'hey, it's the NBN, I think I will now spend $70 per month on Internet access'.

That is going to put most users on either a 12Mbps or 25Mbps circuit speed plan type - for Exetel customers - or less, if you take the industry average of published prices.

I _think_ that is going to work out at an average peak speed of 14-16Mbps. But maybe someone with a better grasp of stats can correct me.

On my way to Sri Lanka this weekend, I had an overnight stop-over in Bangkok. What I noticed this time more than others is just how good the Thai people are at the art of the 'up-sell'.

It was amazing, I don't know why I hadn't noticed it before. Just about everything that starts off with a price of 250 or 350 Baht ended up with me parting with the better part of 1,000 Baht. And being very happy to do it.

Here's one example; With a couple of hours to kill between hotel check out time and getting a taxi to the airport, I decided to treat myself to one of the great treasures of that country - a Thai foot massage.

250 Baht for a one hour massage, or about $7 at todays exchange rate - very reasonable.

'Sir, would you like a foot manicure as well? We have lady who very good. Only 250 Baht'

Of course I would, why not, since I'm here.

Then, feeling totally relaxed halfway through the foot massage/manicure 'Sir, your fingernails long, would you like hand massage and manicure? 150 Baht.'

What's that. four dollars more? Sure, please, that would be nice.

'Sir, you like some herbal tea? 70 Baht'

Sounds good, yes, thank you.

'Sir, you like fragrant oil neck massage to finish? 100 baht'

Yes, lovely, thank you.

So there you go, 250 Baht going in is 820 Baht, plus tip, 1,000 Baht going out. But the important thing is, I was really happy to spend the money. The masseuses were friendly, smiling and pleasantly chatting away to each other the whole time. And I came out feeling that merely spending four times more than I expected was an absolute bargain.

Thais - the masters of upselling, because once you find out how good the value for service is, you are always happy to spend more, to the extent you appreciate them suggesting how you can do that.

It made me think - I think most of our customers are pretty happy with the money they spend and the service they get (obviously, because there are very few barriers for those that aren't to move somewhere else). But just how much better could we be?

Three and a half years ago we deployed our first data cache. In that time the software has been upgraded many times, and 'tweaked' for performance many more. The hardware has been upgraded each year. And since April 2010 two more cache servers were installed, one each in Melbourne and Brisbane.

The milestones, at least for the last 12 months, can be nicely tracked on the MRTG graph:



So what can I say about it after 3 years? There have been pro's and con's of the system along the way, but at every point it has proved better to have than to not have when all things are taken into consideration.

The benefit of hindsight (and record keeping) shows the one and only performance metric of any long term investment - the net dollar benefit over time. Simply put:

Cost
+ a little more in engineers and network administrators time
+ some early port redirection issues we would have preferred to have done without

Saving
- 50% of the cost of alternative bandwidth

I can't put any figure on the two costs other than 'something', but we got through it, and didn't ditch the system, so it couldn't have been too bad or too much.

The cost savings though, I can calculate, and it comes out to $499,500 (net, after all costs) over the three years. Not a huge saving, but never the less, quite significant given our very, very competitive market and ultra-slim margins.

Not that there is anything wrong with MPLS either inherently or by the fine vendors whose products support it.

But just why do people, meaning the people making the purchasing decisions, buy it for their company?

Here is a valid reason; a large, national network, where the network owner needs the dynamic, self managed, flexibility of 'control over the entire network routing' that MPLS provides.

In practical terms this means; the Service Provider gives you a BGP session from your CE to their PE router for each location, and you, the network owner/customer, do the rest.

Sound like you? Or do you have a network where the service provider gives you a 'fully managed service'? Or perhaps 'A fully managed MPLS service'?

If you are almost every business in Australia, you are in the second, above category.

So now all you have to ask yourself is 'what difference does the MPLS in the description of the fully managed service I am buying make?'

Because the packets coming out of the CPE to your local networks really know the difference between MPLS or any other method of delivery, don't they? Yes, they are so much more relaxed an happy, and their payload of data is just that much better for it, isn't it?

Excuse my extreme cynicism. It just sickens me to continually hear the snake oil pedaled by the charlatans of the industry in this country, that is bought up by their ever eager audience of non-competent decision makers.

If your packet latency is 20ms between two points, do you really think it makes any difference if it gets there by MPLS, GRE, IPSEC or just plain IP, or any other TLA, FLA or nLA someone might dream up in the future? Of course not. A packet is a packet is a packet. Your data could just as easily be encapsulated on a RAM stick put in an envelope an posted to you. You might notice the latency difference in delivery, but your data sure as hell wouldn't know any difference.

Here's a hint; if you see MPLS anywhere in the proposal your service provider gives you, and you don't know what a CE or PE is, then the only thing MPLS means to you, is you will be paying somewhere between 200% and 500% more than you need to for your circuits. But don't worry, your CFO might not find out your blunder this year, or maybe even next year - and who can think further than that into the future.

Oh, incidentally, do Exetel use/sell MPLS? Of course we do. Our routers and network uses the same sort of equipment and OS's as everyone else's. There are some very good reasons for us to use MPLS in our core and connecting to some of our suppliers. It is also readily available for our customers. But, I have to tell you, to date, for every one who as asked for MPLS (always on the basis of 'their last supplier did it, or 'we must have it/no, I don't know why, we just must'), we have always found a more efficient, cost effective way to deliver the same services without MPLS.

To: Exetel Staff

Ladies and Gentlemen,

I have mentioned it to a few of you, and I think it only fair to tell the rest of you first hand.

My New Years 'resolution' was that 2011 would be the year on 'now' as far as Exetel is concerned.

Over the years of our operation, I have seen one of the founding principles on which we based the company slowly, slowly, slip away. So slowly in fact that I realized, eventually, after thinking back, that even I had become complacent with the downhill slide.

So, that is what 'new years' are good for - you can make a decision that, whatever happened in the past that you weren't happy with, you can take action now to stop it making you unhappy in the future.

And so I have.

The principle to which I refer in this case is the immediacy in which things were always done. That is; every email or request to anyone elicited an immediate response; any request for a change had an immediate response of:

- yes, it can be done, and I am doing it now, or,
- no, it can't be done as you ask, and here is what I suggest would achieve what you want, or
- in the rare cases it is warranted - I am not sure I fully understand, can we please discuss it (and then I will do it right away).

Everyone in the modern world talks about reduction 'stress' in the workplace, and a better 'quality of life'. I can tell you with the absolute certainty of long experience and careful observation over many years, that once you have a place to work that stops the rain coming in, and an income that pays your essential bills, the ONLY thing left in your working life that creates any stress at all is your dependence on your co-workers to respond reasonably to your requests.

The Directors of Exetel do their very best to address the first two of those factors - as much as any business of our small size in a very competitive market can do.

Only each of you, making a decision in your own mind, can to anything to address the one, remaining, very large factor.

Think about it:

- What do we hate most about our suppliers? They take ages to respond and only sometimes meet the dates they have set.

- What do we despise about our competition? We are told time and again by customers who have moved to us that we provide fast, accurate responses to questions, that they never got from their previous supplier

- What do we loath about our contemporaries in the Industry? Their acceptance of slack practices in their own organizations means it takes is days or weeks to get any information we have asked for.

So you can see the common thread. As soon as one person accepts that 'by the end of the week' is an ok time frame, then the person they need to give the answer too has to be told 'by the end of the month', which practically equates to; nothing you want to get done can be done in less than a year, but probably more.

We hate those qualities in others, we should never, ever, accept them in ourselves.

There is only one time frame to do something, and that time is now. This is the year of now.

Regards,

Steve

ps. If you are a manager or supervisor, it is part of your job to preempt what might be asked of you. It is 100% not your job to be asked to deliver something, and then suddenly have to have a lot of meetings and take weeks or months to come up with what you have been asked for. Sometimes, even the best of us will be caught flat footed, but that is when your qualities as a leader will shine through, and you will make a response based on the facts you have to hand right now, and the best of your ability.

I was ask to reply to some emailed questions from a freelance journalist the other day, who asked:

Q. > I wanted to look at it from the perspective of ISPs, who will be primarily responsible for providing these types of services, and understanding the possible commercial return.
>
> - Are you considering offering services such as IPTV, videoconferencing package, or e-health services? What would this deliver to customers and how much would this cost?

A. We have offered IPTV in the past, and IPTV is currently actively offered either free or at a small cost by some ISP's now. When offered as a subscription service, even at a very small ($2 per month) fee, it did not prove very popular. From a base of 50,000 potential users, the take up rate was less than 0.2%.

To be of practical use, IPTV requires a set top box, or a computer dedicated to that purpose, or a method of retransmitting data from a PC to a TV that is not affected by any other use the PC is put to (and that the PC is powered on all the time). The latter two methods tend to be 'too much bother' for a user not otherwise interested in the technology, while a set top box adds extra cost and places the IPTV service squarely in competition with satellite/cable TV and TIVO type services - which IPTV has a great deal of difficulty matching for interesting content at a similar price.

There are a large number of commercial video conferencing product available on the market now. So too are there very many personal products available - as can be seen using any chat service (MSN, Google etc).

Since the NBN is not being built as a commercial grade service, it is unlikely to be suitable for commercial video conferencing purposes (for example, the shared transmission method will not allow the QoS controls needed for a commercial grade service, and possibly there will be security issues). Which in any event are available over, and widely used on, metro Ethernet and SHDSL type services. We have many customers, and we ourselves use this in our business. The cost is much more a factor of the level of picture quality the user wants, and the amount they are prepared to spend on equipment.

As a 'home user' orientated service, the video conferencing therefore falls into the private use category - which is addressed very well by current broadband products in the $20-$50 range. (you may even recall before broadband, millions of people still used video chat quite well on 56k dial-up).

I am not sure how you are defining e-health. But in general, the same is going to apply as for video conferencing. I would think privacy and QoS issues would prevent the NBN from being used for doctor-to-doctor or doctor-to-hospital consultations. I suppose if every household also installed $20,000 of commercial grade high resolution audio visual equipment, then it would be possible to use the NBN for doctor-to-patient consultation - except were tactile examination is needed. But that could also be done with ADSL services as they exist now.

Q. > - What is the market demand for these services now and how will this look in eight years time when it is proposed that the NBN will deliver 100mbps internet connections?

A. Our 'toe in the water' site in Tasmania indicated that when offered for free, the demand for NBN is less than 1% (in balance, the sample size is small and the demographic may not be indicative of other areas). Never the less, it is a matter of fact that all households in the NBN footprint already have access to bandwidth in excess of 100Mbps for broadcast services via cable, satellite or free to air TV. I think in eight years time the Internet, whatever the media used, will still be an expensive way to deliver those services.

However, it would be a total Luddite that didn't think that over a decade 'something' wont come along that will require a) much more bandwidth, b) requires real time bi-direction data, and c) is a killer app everyone will want.

But that has to be balanced against:

- improved compression methods and greater processing power, requiring less bandwidth
- improvements in electronics, providing greater bandwidth from existing copper and wireless
- increase of mobility devices (eg iPad, iBook etc) and greater requirement for portability

Which will make a fixed line product less relevant to the user base, and comparatively more expensive

Q. > - Is it commercially viable to offer these services now? If not, what market conditions are required for this to happen?

A. People are using all services the NBN offers now on ADSL and 3G/HSPA. Any ISP that makes their financial information available will show if they are commercially viable or not.

Q. > I'm keen to hear your thoughts on this topic and how it can be explored.

A. The NBN, in my view, is essentially a mechanism to remove the Telstra monopoly of copper to every house. Maybe it is worthwhile for that reason alone. Or maybe what could be explored is a more cost effective way to do that.

Review time once again; that all too important exercise of, formally, asking each person within the 'umbrella' of your responsibility how they are going, how would they like to be going better, and how long they think they will keep going for. Because, what else does anyone really need to know other than that?

The 'traditional' review questions along the lines of 'targets/performance' are discretely measured along the way. Certainly in terms of any engineering/systems function, it is very hard to argue with the minute-by-minute data from the monitoring system. Nor can the goal of 'zero faults' be argued against the actual achievement of the monitoring reports that everyone has the same access to.

They change a little from time to time, but these are the review questions for this quarter:

1. Where your job goals met for the previous quarter?

2. What can be done to improve your current duties?

3. What would you like the me/the directors/anyone to do to help you with your career?

4. How long will you stay with Exetel?

5. What do you like most about your employment with Exetel?

6. What do you like least about your employment with Exetel?

7. What are your job goals for the current quarter?

8. Anything else you would like to discuss

So, six questions to find out the three most important things. One question to checkpoint that the 'automatic' metrics are working, and one question to catch anything else anyone feels is important. I have no idea how that agenda would fit in with an HR Managers view of how things should be done. But then, I have been to HR review meetings, and the only thing I ever got out of it was the impression the HR people found their job pretty depressing, and we were both glad when the mandated 90 minute meeting time was up.

The review questions seem straightforward to me - which is exactly how they are intended to be. Never the less, I have never held a review where I haven't been surprised by responses to one or more of the questions - and that is always where the most value of the exercise comes from.

Speaking of exercise, I took a couple of days leave during the school holidays and went camping in the Stirling Ranges with my two younger sons. For a continent as geologically stable as Australia is meant to be, I was surprised to find how much steeper and higher those mountains have become in the 25 odd years since I last scaled them.

Our campsite was to the west of Bluff Knoll, and in the very early morning the valleys are in darkness while the peaks literally glow from the rising sun. It only lasts about 15 minutes, but we could actually watch the terminator sweeping the wrong way from west to east as the sun rose over the eastern mountains behind us and the shadows retreated down the valleys to in the west.

We have our very first users connected. Qdos to Paul and his team in provisioning, and Brendon and his team in F&A for their truly heroic efforts of perseverance in the face of, at times, seemingly overwhelming bureaucracy.

I can hardly wait now until my suburb is connected - only two years at most (or whatever the latest national roll out plan is).

I was having a think about all the things someone will be able to do with the massive bandwidth available through the NBN, and comparing that to what can be done now. This is what I can up with:



I don't know, maybe I have just become way too cynical. All those 'field of dreams' networks that were built in the 98/99 era got used, eventually, with only two or three successive owner bankruptcies along the way.

It's late, and I'm tired. Maybe someone else can come up something it would actually be useful for.

(yes, I meant to put Qdos. We all remember QDOS from Seattle and later Sinclare, right?)