Steve Waddington's Network Notes

Great. No sooner had I hit return on the last post, when 5 sms threshold alarm arived. BGP is flapping on all border routers, caused by _something_ - no one knows yet.

Latency to the Sydney servers from where I am is 1.2 seconds with 40% packet loss. Totally unusable to access any of the routers and investigate or assist directly with working out what it going on.

I can get email after a fashion, so I can see the messages from customers hitting the forum, but the connection is too useless at the moment to even access the forum. Not even sure if this blog post will make it.

Ah, here we go, pings are back so 120ms. Whew. No I just need to wait for Vinna's incident report email.

What a horrible start to the week.

On my way to Colombo to work with James there to get our comms and other bits and pieces sorted out.

Reading on the plane last night I saw this old quote from Arthur C Clarke "Any sufficiently advanced technology is indistinguishable from magic." (Clarke's Third Law of Prediction).

At the time for some reason I was also thinking of the Fermi Paradox, which Clarke alluded to and Baxter stated outright, isn't a paradox at all, rather, the evidence is all around but we don't know what we are looking at.

Which led me to the thought that perhaps any optimally advanced technology is indistinguishable to nature?

Maybe it could be stated as the Clarke-Baxter nth law:

"Any optimally advanced technology is indistinguishable to nature"

So why aren't these optimally advanced beings enlightening us so we don't have to continue to wallow in the mire of our own human stupidity? Ah, well, you see, they are. The plain and simple message is bombarding us all the time, across every spectrum. But of course:

"Any optimally encoded signal is indistinguishable from noise"

We have the answer then, all we need is a way to decode it. Tricky. I'll have to think about it. The program will take a little while to run. Maybe Douglas Adams knew, but he's not telling.

An interesting article here in PC Magazine. In the past I have termed the US the 'land if infinite, free, bandwidth', a sort of semi-joke referring to the far far greater backbone capacity, and far, far cheaper cost (for non tier 1 carriers) of access in the US.

Last time I looked (or rather, noticed an email about it on NANOG), bandwidth was being offered for $10 per mbps to a small regional ISP who only wanted 50Mbps. One can only assume that larger quantities would attract much lower pricing.

But let's stick with $10 per Mbps for arguments sake. Even though the price of bandwidth in Australia continues it lifetime trend south, even at the best rates I have heard being offered today, we are still in the order of 15 times that cost.

For Comcast to limit users to 250Gbytes per month must be indicative of some stress, either on the bandwidth or routing capacity of the network, and their engineers provisioning forecast against plotted growth of usage increase. Of course, I could have it completely wrong, but it has to be something like that.

So in todays Australian cost terms, it is like and ISP constraining downloads to 16Gbytes per month. Which, when you think about it, it around the median of broadband plans on offer anyway.

I sort of hoped when I started writing this, that some profound revelation would 'pop out' by the time I got to the closing paragraph. But is hasn't. All I can think is that end user bandwidth use in the US has finaly caught up with the infrastructure build of the dot com eara, and no can can see where the investment are coming from to fund the next quantum leap in capacity.

We have been testing HSPA access in preparation for going live with our own HSPA product. From the initial testing I have done, the results indicate to me:

- Signal strength is no indication of throughput
- there appears to be a good deal of traffic manipulation going on within the carrier network
- Ping and traceroute results do not correlate strongly with download performance
- VoIP is usable even when ping return times would indicate otherwise

Here are some examples:

- From my home in a southern suburb in Perth, I was getting 95% signal strength. Over the period of the morning, ping times went from 350ms to 800ms back to an Exetel server. Download speeds at that time were around 1.9Mbps. In the evening, ping times were as low as 180ms and averaged at 300ms, however the download speed was around 800kbps.

- From a cafe in Perth city, I had signal strength of over 90%, ping average was 390ms and download speed was 800kbps. However from Clancy's Fish Pub in Fremantle (being and HSPA tester is hard work), pings averaged 420ms. signal strength was 45% but download speed was 1.5Mbps.

In all cases there was the odd ping result of 1,000ms +, and an occasional packet drop. Despite that, VoIP from any location was always about the same, rating a MOS between 2.2 and 2.6 each time. (I was using a cheap gaming headset/microphone combo with X-Lite from a laptop. I suspect a better quality setup would easily add .5 to the MOS score).

A typical traceroute would show almost all of the latency occurring on the first hop, and quite often the latency improved through subsequent hops until it left the carrier network. For example, the first hop from my laptop might show 420ms, hop 2 470, then 398, 418, 416 etc and then 450 on the final hop. Considering the first hop is, presumably, to a local node in WA, and the carrier network termination is in Melbourne, I surmise that most of the latency is induced due to traffic policing within the carrier network.

Which is fine for TCP and ICMP, which are not latency critical, at least for most business applications. Gaming I would expect would suffer though.

On the other hand, UDP and RTP's seem to slip through the policing cordon, because the VoIP results were essentially the same as my 'control' which was using the same laptop and headset but from my 8Mbps ADSL1 line at home - where my ping is a consistent 90ms to the Sydney SIP server.

We have 10 test units out in the field, and my test alone is not enough to be a representative sample. We should be getting all the results back in the next week, so it will be interesting to see if my preliminary observations hold up.

I mean, what went wrong...(statement leading to answer, rather than a question).

When things that should go perfectly smoothly go wrong, I can't help but be reminded of my high school science teacher,. Mr Savil (a very competent teacher and electrical engineer in his own right), yet who's every experiment in front of the class seemed to defy the laws of physics, thermodynamics and chemistry, and somehow go wrong (often filling the classroom with acrid smoke causing evacuation and the rest of the period off).

Anyway, no smoke came out of the 6500. The physical changes were done by 5:15am, and then it was just a matter of making a few route path changes. Which were in turn completed by 6am.

Everything was looking good route-path wise, and the traffic through the 6500 was matching expected levels. However by 6:30am we could see these problems:

1. Traffic into the network was only 700Mbps, where it should have been 1.6Gbps

2. There was odd packet loss between the 6500 and the ESR LNS's, and much higher than expected latency to the second Sydney POP - 200ms where it should have been 12ms

3. A check of the 6500 CPU load showed it fluctuation from 8% (about what was expected) to 95% over a period of 15 seconds.

The plan was to commence roll-back at 6:30 is things hadn't worked out. The problem was though, that as far as the change plan had gone, and the possible issues we thought we would encounter, everything was spot on. The odd issues we saw (we thought) were probably due to some some simple cause, such as a mis-balanced data path causing an interface overload, that could be quickly fixed.

So we delayed the rollback and went looking for the possible error. We actually found two, and with a sigh of relief, made the load balancing right and volla now everything works...

not.

Whatever the problem was, it was not anything obvious. It was 7:20am, and while the deployment team were telling each other what problems we could see, none of us had an answer to how to fix them.

I gave it a few more minutes, but at 7:25 had to order the roll-back.

Very frustrating.

The roll-back took us a little over our maintenance window, and was complete by 8:20am. Although later we were to find some other routing issues that has caused loss of access for some netblocks to some servers.

We broke for coffee at 8:30 and with the main time pressure off, we discussed what, possibly, could have cause the problems we saw. We considered a number of speculative possibilities, but our main consensus came down to some sort of IOS issue. Looking at the traffic graphs between the time we had the routing 'good' and the roll-back, it was apparent that the sum of all traffic inbound to the 6500, which was across 4 GE circuits, did not exceed 960Mbps - where it should have been around 1.9Gbps. Those graphs were suspiciously all flat lining at just under 300Mbps. Which seemed to indicate that there was a 1Gbps circuit bottleneck somewhere in the traffic path. Going through the connections and traffic paths step by step, it is still just not apparent what could cause such a thing.

This one will have to go to Cisco TAC for some deeper diagnosis. And consequently, there was no point continuing the work on the planned Sunday maintenance window, because without knowing the cause, were were bound to strike the same problem again.

A disappointing result.

The biggest single change we have ever done on the Exetel network is in its final stage of planning now. We will be deploying our new 6500 router to replace three border and two core routers in our Sydney POP.

The change itself is fairly straightforward; the interfaces and functions of several routers are easy enough to duplicate on the 6500 router, and we can stage the change over - circuit by circuit - so we can see early if anything goes wrong.

We took delivery of the 6500 three weeks ago, and since then Vinna has pretty much exhausted all the testing and preparation combinations we can think of. Vinna's deployment plan looks sound to me, covering what should be a straightforward change and the likely possible problems that may eventuate. And independently I have covered all the more wild and implausible possibilities I can think of.

Yet, one must never become complacent. It is perhaps an instinct I have developed over the years to let a feeling of trepidation build up that something _may_ have been missed, to further spur my imagination to come up with just what that might be. It makes for some sleepless nights, I can tell you. But if you have ever had that terrible feeling when a catastrophe strikes and you don't know how to deal with it, then you will understand why a healthy dose of paranoia before the event can really pay off.

This particular 'operation' will take place over two days (two early mornings rather). First, the 6500 will be sat next to the existing routers, and one by one, the cables swapped from the 7xxx interfaces to the 6500 ports. That part should take about two hours, so starting at 4am, we should be done by 6. However, the best laid plans etc, so the window is from 4am to 8am, just in case. Should the worst happen and nothing works, roll back will mean just moving the cables back to the old routers, which are still in place and 'hot'.

At 6:30, if everything is ok, checking will be complete and the deployment team can hand over to the monitoring team and stand down. The monitoring team's job is to keep an eye on everything for the next 21 hours. If all is still well, the deployment team will return at 4am Sunday to:

- power down the now disconnected 7xxx routers
- remove them from the rack
- install the 6500 router in the rack
- reconnect and double check its operation

Which will complete the work. Again, 2 hours should be needed, but the window will be for 4 hours, just in case. Then we can all go and have breakfast (nearby). Check again, and we should be ok to let the normal monitoring systems take over after that.

In a way, it will be like a heart-lung replacement, but with one big difference. As bad as it would be, the cardiothoracic surgeon can always lose a patient. In our case, we have no such option; our patient CAN NOT die, or indeed, the time of great tribulation will be upon us, not to mention eternal damnation.

I think we are ready. A bit more checking and then we post the maintenance notice.

Yea, though I walk through the valley....

I read this article with, frankly, not much interest. Same old tired rhetoric.

"Your users stole our copyright material" blusters AFACT

"We aren't the police" Whines the ISP "An besides, it's impossible to do anything about anyway"

Well, the fist part of the ISP's statement is certainly true. Although the second part is either made by someone ignorant of what is technically possible, or someone hoping their fib will be believed by someone who is technically ignorant.

As far as not being the police goes, and no one seems to have spelled this out so far, the ISP's dilemma is this. 'Someone' claims that a customer of the ISP is stealing something of theirs, and they request that the ISP prevent the customer from doing it any more - namely disconnect the user.

However, the ISP has a contract with the end user to supply a service. Can the ISP unilaterally cancel a service on just the say so of someone else. What legal right does the ISP have to do that, and what legal right does the 'someone' have to demand that the ISP does that?

Sure, the 'someone' can present whatever evidence they have to the ISP. But what ability does the ISP have to make a judgment on that 'evidence'? AFAIK, in Australia it requires a court to make judgments on those sort of things (in fact, right at the moment we are spending a lot of money on legal advice to find out).

But why is the ISP in the middle anyway?

If someone is drunk driving, no one goes to the MRD and demands they stop that driver from using the roads any more - with the implication that if the MRD doesn't take action, they somehow also share the blame.

If someone, say, plots a bank robbery with their nefarious mates using mobile phones, no one goes to the phone company and implies they are an accessory to the crime for not preventing the network from being used that way.

Yet those are the outrageous implications made in infringement notices sent to ISP's.

And, it seems, until it is tested in a court of law (at a cost of most likely not less than $500,000 for the ISP), no one will know what the actual case will be for ISP's.

However, look at it from the agents acting for the copyright owners point of view. They see someone that they believe is stealing their clients property, and it is their job to act in their clients interest and stop that. How can they? All they have is an IP address that, when they look up who it belongs to, tells them it is some ISP somewhere in the world.

What are they to do? Write to the ISP and request the contact details of the user of the IP address? Ok, but in any country with privacy laws, they will most likely either be ignored, or at best, told that the information they seek can only be obtained with a warrant (which I believe is the case in Australia).

Then where does that leave them? Pretty much they have to say that 'well, we have told you we think one of your customers is stealing our clients property, you wont tell us who that person is or do anything about it, so we have to assume you (the ISP) is complicit in that theft.

Nice one. The ISP can't do anything because is breaches their contract with the customer and privacy laws. The Copyright agent can't protect their client because the ISP wont (or legally can't) give them any information or take action to further prevent it.

One thing is for sure; the lawyers on both sides can start booking their next family vacation in the Swiss Alps.

But maybe there is a solution. A pretty simple one too.

The complaint is between the copyright agent and the end user. The ISP is only the 'meat in the middle' because privacy laws seem to say end user information must not be divulged.

But surely if you think someone is stealing from you, you have a right to know the identity of that person? In most cases, if all you had was a phone number, it would be a trivial exercise to look up the number and find the persons name and address in the phone book. Why should not the same be true for IP addresses? Why should an IP address be any more 'private' than, say, your street address? It's the same thing - globally unique, one is where you live in the world, the other is where you 'live' in the Internet.

I remember an old credo of the Internet 'Information wants to be free'. If that is indeed the case, then it can only work if it goes both ways.

So one solution is; let the copyright agents have the contact details of the person they have the complaint about. Such agents are always lawyers themselves - and hence officers of the court. Is there a difference in handing over end user contact information to them, rather than to, say, a debt collection agency?

Of course there is a difference, spelled out specifically in the Communications Act, as our own lawers have just pointed out to us in response to our request to them for advice:

Section 276 of the Telecommunications Act provides:

276. Current eligible persons

(1) An eligible person must not disclose or use any information or
document that:
(a) relates to:
(i) the contents or substance of a communication that has been carried by a carrier or carriage service provider; or
(ii) the contents or substance of a communication that is being carried by a carrier or carriage service provider (including a communication that has been collected or received by such a carrier or provider for carriage by it but has not been delivered by it); or
(iii) carriage services supplied, or intended to be supplied, to another person by a carrier or carriage service provider; or
(iv) the affairs or personal particulars (including any unlisted telephone number or any address) of another person; and
(b) comes to the person's knowledge, or into the person's possession:
(i) if the person is a carrier or carriage service provider--in connection with the person's business as such a carrier or
provider; or
(ii) if the person is an employee of a carrier or carriage service provider--because the person is employed by the carrier or provider in connection with its business as such a carrier or
provider; or
(iii) if the person is a telecommunications contractor--in connection with the person's business as such a
contractor; or
(iv) if the person is an employee of a telecommunications contractor--because the person is employed by the
contractor in connection with its business as such a contractor.
...

(3) A person who contravenes this section is guilty of an offence
punishable on conviction by imprisonment for a term not exceeding 2
years.

Such conduct would also be likely to breach section 7 of the Telecommunications (Interception and Access) Act 1979.

7. Telecommunications not to be intercepted
(1) A person shall not:
(a) intercept;
(b) authorize, suffer or permit another person to intercept; or
(c) do any act or thing that will enable him or her or another person to intercept; a communication passing over a telecommunications system.

Personally, as much as I don't like the idea of someone using the Exetel service to steal someone else's property, I much more don't want to go to jail for 2 years for giving that alleged thief's information to someone else.

So it seems what is needed is a streamlined and automated 'warrant' system, whereby the copyright agents 'swear and avow' (under some threat of terrible reprisal) as to the strict reason they want the information and the purpose they will put it to. Secure certifications means that such information could be sought and obtained from the ISP with a high degree of automation - if not in real time, then pretty close to it. And the very great benefit, to both parties, is that from that point on they are in direct communication with each other, without, necessary legal, interruption of a third party ISP.

John asked me 'What value did you get from the conference last week?'

As I mentioned, the event exceeded my expectations. I was expecting to maybe see one or two things 'of interest', and maybe pick up another one or two points of some value from the presentations. But there was far more than that.

After a review of my notes, the nine things of greatest value were:

1) Allot recognize the issue of mis-classification and have two ways it can be fixed. One is by turning off the classification memory, the second is to upgrade the service controller server.

2) It was clear that every service provider, including major regional Telco's and every major Australian service provider are using DPI of one sort or another to manipulate traffic

3) From Taiwan Telecom I found out an different way P2P policies can be set up in a network using TOS settings

4) There will be a 2-3 year period before DPI, at least from Allot, becomes a 'mainstream' product - however, they are probably further ahead or equal with other DPI companies

5) DPI alone is not an optimum traffic control solution, DPI + Caching leverages both technologies for much better results

6) Allot have released a new chassis based server, that can include a cache blade and also an intelligent network protection blade

7) Allot have acquired a company that makes a network protection server - it detects things like spam and bot-nets from inside the network and can automatically isolate and inform the end user. It also detects attacks from outside the network and can shut them down

8) We will almost certainly need to move to 10Gbps in out network core within the next 12 months

9) There is a low end Allot product we can use for Brisbane and Melbourne, and there can be unified policy control via the service controller server

Plus it was very interesting listening to what companies Like NTT, China Telecom and Korea Telecom had to say, as well as the other 'smaller' (but huge by Australian standards, having more subscribers than our entire population) service providers from India, Indonesia and elsewhere. And interesting to see that as an 'early adopter' we are ahead of almost everyone, if not everyone, in the way we have deployed and integrated DPI and caching into our network.

Between the conference ending and our flight home, we had a free afternoon in Bangkok to do some sightseeing (read; 'shopping'). Before consigning myself to the roll of bag carrier for the afternoon, I particularity wanted to visit a gem shop that had been recommenced to me by friends who have brought some very lovely sapphires there.

Looking at the tourist map of the city, the address we were heading for was about two blocks north of the Shangri-La Hotel - an easy landmark to find. Never the less, we first stopped at the concierge desk to just check (read; allay the fears of my travel companion) that my infallible sense of navigation is foreign cities was in fact infallible.

'Yes' the nice lady at the desk told us, after looking at the piece of paper with the address 'I know that street, just take the hotel shuttle boat to the MRT landing and it is about 15 minutes walk from there, or you can get a taxi'.

We got to the landing and I was looking around for a way to cross under the sky rail to get to the Shangri-La, from where I would better know the way - not the quickest route to our destination, but one I could be sure of.

While doing so, no doubt looking like clueless tourists, a friendly lady came up to us and said something like 'Hello, can I help you, I work at the hotel (the Hilton where were were staying) and I recognize you from there.'

Barbara started talking to her and they chatted for a minute about where were were from...

'Australia - Sydney?'

'No, Perth'

'Ah, my sister live in Perth, she married to an Australian there, I go there next year'

... while I was trying to refold the map to some manageable size.

'Where you looking to go (sorry my English not so good')? She asked

'We are looking to go to this address' I said, showing her the piece of paper.

'Ah, you looking to buy gems?' She asked, obviously recognizing the street or area where all the gem shops are.

What followed was about 10 minutes of very polite conversation of where to get the best gems, her showing us a beautiful star sapphire ring she was wearing and some other jewelry she had on. She went on to explain that were we were headed was really only for tourists, and the prices were higher. But she brought her gems from a government wholesale outlet, much cheaper. What is more, the quality is strictly controlled and guaranteed by the government.

'Well, we are going to have lunch first, and then we may go there if we get time' I said, it being about 1:30pm

She went on to comment that today, for the Kings birthday, the government was excluding sales tax on gems. But that the government wholesaler would only be open until 3pm, and a Tuk Tuk could take us straight there, only 20 minutes away, but too far to walk, only cost 50 Baht . She then proceeded to write down the address for us, somewhere along the way asking if we spoke Thai. No, we didn't. so she kindly also wrote the directions in Thai that we could show the Tuk Tuk driver.

We thanked her and went on our way - toward the Shangri-La to recommence my navigation to the place we originally wanted to go.

Not more than 20 paces on, as we were heading towards a footbridge that looked like it would lead to the street the hotel was on, I must have showed some further clueless tourist hesitation, because a well dressed gentleman came up to us and asked where we were heading.

I told him were were looking to get some lunch. He hesitated a moment, and by way of friendly introduction told us he was a doctor and his office was just nearby. 'Nowhere good here to eat' he told us 'You need to go into city for best food, come with me, I show you way'.

The way turned out to be a taxi rank nearby, where our newly acquired guide told us he would tell a Tuk Tuk driver to take us to where all the nicest food was. He was a dentist (a doctor dentist apparently), so he knew about good food. While he was instructing the Tuk Tuk driver where to take us, I told him we were meeting friends anyway, but thank you, and very nice to have met you, and we walked off.

Barbara asked me why we didn't take the tuk tuk to get some food (it was now about 2pm and we had both skipped breakfast and so were pretty hungry). 'My dear' I explained 'We were 100% being scammed'.

She didn't believe it. And why would she. Thai people are very friendly and commonly will help out a foreigner who looks a bit lost. The first lady who spoke to us could well have been from the hotel. She was nothing but friendly, polite and at no time was their any pressure, other than her hospitality, to follow her suggestion. So too was the doctor/dentist very polite, well dressed and well spoken. There was no apparent connection between him and the lady, they were just two people trying to be friendly and help out some tourists. Everything they said was completely plausible.

Never the less, with my now slightly grumpier companion (the address we were looking for was both farther and more difficult to get to, in the sweltering midday tropical heat, and we never did end up having lunch), we eventually made it to the address I wanted. And, incidentally got a fantastic price for about nine carats of gorgeous sapphires.

Still discussing on the way back to the hotel why I had declined the advice of those very nice people, I cited as reasons the coincidence of the lady just happening to know the 'best place for gems', the coincidence that, after asking where we were from, she just happened to have a sister who lived there. The coincidence that there was a special discount for the Kings birthday, but we had only until 3pm (not enough time to go back to the hotel and look it up on the Internet). The 'safety' that the gems were government guaranteed (when if fact there is no such thing). The way the lady made sure she asked us if we spoke Thai before she wrote, what _could_ have been the directions in Thai. The 'doctors' slight hesitation when we said we were looking for a place to eat, rather than how to get to the government gem outlet. And then how he became a dentist, and steered us to the taxi rank with the one, conveniently waiting tuk tuk.

But all that was in vain. In the end I had to resort to showing her the web sites that explained exactly how such scams work - almost word for word, that I had read before we left Australia. Just google 'sapphire scam', there are at least thirty sites that expose what can happen.

Quite frankly, to me, the whole event was highly entertaining - in fact, the highlight of the trip. The sting was almost flawless, well executed, and completely plausible. You just can't pay for entertainment like that.

Henry Kissinger wasn't there, but just about everyone else was. I am talking about the 'Allot Broadband Service Optimization Summit 2008'. Allot of course make the NetEnforcer product that does deep packet inspection, analysis and control of IP traffic.

I have to say that the event was exceptionally well run, far exceeding my expectations in every respect. Not the least of which was the amount of information packet into the, essentially, two day event. First of all, the list of attendees was quite imptessive, with representatives from telco's like China Telecom, NTT, Thai Telecom, TNZ, Singtel - in fact all the largest telco's in the Asia Pacific region were there.

Another interesting thing was the Australian ISP's who were, and were not there. Including one that was 'incognito', apparently, I can only guess they don't any anyone to know they are manipulating traffic. Talk _is_ cheap, and 'high performance internet' extends only as far as your DPI box, right guys?

In casual conversation over breakfast I was chatting with well connected industry person about the notable absence from the Aus ISP community. Not because they don't have DPI and traffic shaping, I was told, but because they have bought a competitors (to Allot) product (and I place some veracity on that information, because the person I was talking to has been marketing Allot to those companies and is in a position to know what they have).

So that is a long way to get to the point, which is - EVERY Telco and ISP (of any note) is, right now, doing DPI and traffic shaping. Whether they tell their customers or not.

The second great thing about the summit was the presentations. Very senior people from seriously serious Telcos, spoke about their DPI deployment, issues and experiences. Needless to say, a common concern expressed by everyone was the 'P2P Problem', which is twofold.

One; it can saturate even the largest most robust network.

Two; it is by far the cheapest way to distribute content. How cheap? I'm glad you asked. These figures are from Equinix to compare the cost of methods of content distribution (by the content owner) for 10 feature length videos (I assume the cost is US centric):

1. Transit (just buying enough bandwidth)......$0.60c
2. Content Distributed Network (Like Akamai).....$0.77c
3. Transit and CDN hybrid.....$0.69c
4. P2P.....$0.18c

Cheaper by a factor of at least three. As you would expect, because the end user is the one who 'pays' for the distribution bandwidth. All the distributor needs is something popular enough to inspire seeding. I can only imagine someone like Microsoft would save an absolute bundle on their massive server farms and bandwidth if, say, XP updates were P2P seeded. But that is something Linux distro people have known for a long time.

Being the cheapest distribution method isn't a problem to the distributor of course. But what is means to the ISP is that P2P traffic is only going to increase, and increase dramatically according to all the nice graphs annalists show, in the next five years. Further taxing networks and bandwidth. And while bandwidth can always be increased, the expense and time of 'raw' bandwidth just wont keep up with demand.

The solution (or perhaps selling point) is, surprisingly, not just DPI and shaping. Because what that does is improve the QOS, but at the expense of degrading the 'Quality of Experience' - QOE. What is needed therefore is a way to manage network bandwidth and protect it from P2P saturation, while at the same time delivering good 'QOE' for access to ever increasing high bandwidth content.

Enter the combination of DPI and P2P Caching.

No surprise that Allot and PeerApp recently announced a joint solution for Video optimization.

The great thing about that for Exetel, is that we will be able to leverage what we started as a 'go it alone' solution nine months ago with refinements Allot and PeerApp develop for the global market.

In fact, the shape of things already here... if you live in Korea.

What KT are doing now with IPTV makes the latest Foxtel Digital offering look very outdated.

So what makes it so impressive? Here we go:

- Currently 68% of KT broadband customers are connected at 50Mbps via VSDL or FTTH. It will be 92% by 2010 (about the time the tender to begin to plan to think about starting to plan for the far more basic FTTN comes out in Australia)

- Providing capacity for UNLIMITED channels via multicast IP

- view any program any time

- Personalised services; essentially, every subscriber can create their own customised TV channel

- fully interactive services on the screen, including; news (a bit like google news - with the topics of interest to the subscriber), SMS style messaging in message pop up on screen, phone callse and CID pop up on the screen, share trading, shopping (with fully integrated product and best deal search)

- search engine like search for programs, news items, products, information etc.

- custom services for businesses such as in company broadcasting

- home media IP hub; service not just available on one or two TV screens, but to any networked device in the house

- Pop up information about the actor, or the car he/she drives, or the product he/she is using in the program

- real time on screen polling and feedback

(of course, just imagine the information that gives marketers)

- tracking of viewing habits and preferences to make intelligent suggestions of programs you might like to watch

And the absolute coup de gras killer app.....

- Watch sport from the view you select (changeable any time via a sidebar on the screen), or from the view point of your favourite player, or follow your favorite player through the match.

Whew indeed, if the 'whistlers' in the Foxtel ad saw what KT are doing, their lips would fall off.

To smoke? When the government goes to such great lengths to tell us how bad it is for us. But hey, who can really trust what the government says.

To drink and drive? If I just take it easy through the back streets, I'll be right. Besides, I'm too wasted to walk.

To blow your rent and food money at the Casino? Well, I _could_ get lucky.

To not notice your boogie board had suddenly doubled in weight and volume. I er, well, er, umm. Baggage handlers done it!

To steal copyright material and/or distribute it over the Internet. Ah, now, there I am safe, because on the Internet I am completely anonymous and no know knows what I am doing, so I can never get caught.

Of those five statements, it's hard to pick which one is stupidest.

Yet, unbelievably, people will argue that sites with names like 'piratebay' and 'istole-it' are only used for completely legitimate reasons. Of course they are, and you are as pure as the driven snow, right?

Sure, those sites themselves contain none of the, irrefutably, copyright violating material their web sites list as available. We all know how p2p works. But it is ridiculously easy to track and log and identify the IP addresses of anyone who accesses, shares or redistributes that material - through exactly the same trackers that make it available in the first place.

Say you are the sort of person who, seeing a $10 note fall unnoticed from someones pocket, have a look around, see you are unobserved, stoop and quickly pocket it yourself. Ha! No one saw. The perfect crime.

Say you are that same person, but, your quick glance around reveals; three people are looking at you, and two of them are policemen, you, for some reason, are wearing a sign with your name and address on it; and, oh yes, there is that big spotlight right on you.

"Excuse me Sir, you seem to have dropped a $10 note" you say with a smile, having done your civic duty.

And yet it seems there are some stupid, stupid people, who must think that somehow, no one can see what they are doing just because they are on the Internet and not walking in the park.

How many people? Looking at the copyright infringement notices stats for the last week:

date notif block
20080714 3 0
20080715 21 7
20080716 17 3
20080718 8 4
20080719 6 2
20080720 9 0
20080721 7 3

About 10 per day, out of ~55,000. Roughly 1/5th of 1 percent. If you take that a percentile on an IQ bell curve. what is that, three sigma's out? That's not just stupid, that's borderline between mild and moderate metal retardation.

I am attending the 'Allot Broadband Service Optimization Summit' over the next few days - a grand sounding name to be sure. What I am looking to get from the trip is three things:

1. If it is possible to, and how, we can better optimize our NetEnforcer equipment, particularly with the PeerApp cache engine

2. The plans Allot have to improve the traffic classification algorithms

3. A review of why we seem to have so many traffic classification issues - having been told we are the only ones reporting it, and how we can fix that.

I have requested one-on-one meetings with their lead developer and their Director of Technology, so I am looking forward to some sensible discussions with them.

Perhaps it is serendipitous, but four weeks ago we were approached by Cisco to evaluate their Service Controller engine. and they were kind enough to give is a full briefing of its capabilities. So a secondary agenda will also be to get a feel of how whatever goodies Allot are releasing might stack up to the SCE.

It looks like Cisco are fairly serious about getting into that market. Some of the phrases passed across the table were things like 'very keen to do business' and 'trade in existing 3rd party equipment'. I am sure Allot are aware of that, so it will be interesting to see what their plans are.

... would have to be Optus.

http://www.australianit.news.com.au/story/0,24897,24033460-15306,00.html

Both of the circuits that provide full redundancy for the network cut, I mean, what are the chances of that? It really is bad luck. I guess sometimes these things just happen.

Then there was the time a couple of years ago (can't remember exactly when, or the fine details now), when a similar thing happened to their IP network - both circuits that provide redundancy cut, at the same time. One by a landslide/mudslide and the other by something else I have forgotten now (a backhoe maybe?).

And prior to that, going way back to the late 90's now, I recall a similar occurrence with International IP (both routers in different locations US failed at the same time, as I recall). And another time when something caused loss of the data link between Melbourne and Sydney for, was it 36 hours?, I don't recall, long enough to cause much consternation, anyway.

Still, four instances spread over 10 years, who can really complain in the long run? But what appallingly bad luck that both of the redundant circuits/equipment would fail at the same time on each occasion.

The beneficiary of Exetel's carbon neutralization program is Men of The Trees (MOTT). They have lots of information about the work they do on their web site; in a nutshell, they replant areas of degraded or salt affected land with many varieties of trees native to the area. It is worthwhile reading Barrie Oldfield's articles on the MOTT site for a full understanding of the programs they run.

I liked the MOTT programs during my initial investigation as to how Exetel could offset our carbon footprint because of the holistic approach they take. That is, there is more to offsetting carbon emissions than just turning it into wood, which I hope the rest of this blog article will demonstrate.

To put that in perspective, about two years ago I flew down to Albany (my home town). Over the last 25 years farming land in that area has been progressively 'put under trees', as farmers have leased out their land for the purpose of growing trees, as a commercial crop, both for carbon credits and for eventual harvest for woodchips and pulp.

Having flown over the area as a child with my Uncle in light aircraft, I still had a clear memory of how the landscape looked from the air. So flying in on the Skywest plane it was something of a surprise to see just how much of what used to be green cattle and sheep pasture was now blue/green trees - Tasmanian blue gums I believe they are.

What really jarred however was the way the paler blue gums stood out against the patches of darker native bush. Having grown up, hiked through, camped in and driven all around the south west, those trees just looked out of place to me.

That was the first thing. The second is the impact of mono culture tree farming, and using trees that are best for the eventual harvesting purpose. As I understand it, there is essentially not much difference between a tree crop and, say, a wheat crop, just a much longer time frame between harvests.

To stop insects attacking the trees, pesticides are used. To stop 'weeds' (in this case the normal undergrowth you would expect in a forest) and reduce fire risk, herbicides are used. Like any high concentration of a single species, disease is a threat, so more sprays are used to prevent that.

All in all, the cultivation needed for commercial tree farming can only cut into the overall effectiveness of tree farming has in offsetting environmental impact. Nor can it do the native fauna much good - which are just as unwelcome on a tree farm as they hopping, pecking or jumping their way through a wheat crop.

So you can maybe see what is appealing about the Men of the Trees programs. They replant using either local native trees, or species close to it (in some cases where more salt tolerance is needed) and they use typically 15 to 20 different species in each area.

Apart from the initial spray to prepare the re-plant area (to kill off the thistle, dock and cape weed), as far as I know, no other chemicals are used from then on. The flow-on effect is not hard to imagine - native trees promote a healthy under story, making new 'havens' for native fauna, and the return of other native flora species. Once planted there is no more cultivation needed, which must make for more effective carbon offset.

Of course, it isn't a commercial crop - it is done purely for the environmental benefit. While there is no direct commercial return for the farmer, there is however almost certainly quite a few indirect benefits. Here is a brief synopsis of my own first hand experience:

- the farmer has an erosion or salt problem
- he asks MOTT to replant the affected area
- MOTT undertake the project and provide the labour and the plant stock
- the farmer does the initial preparation and, if needed, fences off the areas
- The MOTT condition for replanting is a 100 year caveat on the land
- soil degradation is stopped and reversed
- topsoil run-off is stopped
- the water table is lowered, reducing salinity

MOTT planting is done over the winter months, by MOTT staff and volunteers. This planting season, I volunteered my own and my families services to help out with the planting. For two reasons 1) as corporate sponsors (and the only Exetel staff in WA), it seemed like the appropriate thing to do, and 2) as corporate sponsors, reading about it on a web site is one thing, but seeing, and participating first hand, is what I really wanted to do.

And so, early on a rainy Wednesday morning Barbara and I roused the kids, loaded the thermos and sandwiches and headed up towards York for a MOTT arranged 'corporate planting day'.























The combination of winter planting, using the potting foot tubes, very good nursery tube stock and appropriate plant selection for the area equates to an 80% or better survival rate of the planted trees.

Working the daylight hours we planted out 4,500 trees - a very satisfying days work.