Author Archives: Stubborn Mule

Sleeping Beauty

For the last couple of weeks, I have fallen asleep thinking about Sleeping Beauty. Not the heroine of the Charles Perrault fairy tale, or her Disney descendant, but the subject of a thought experiment first described in print by philosopher Adam Elga as follows:

Some researchers are going to put you to sleep. During the two days that your sleep will last, they will briefly wake you up either once or twice, depending on the toss of a fair coin (Heads: once; Tails: twice). After each waking, they will put you to back to sleep with a drug that makes you forget that waking. When you are first awakened, to what degree ought you believe that the outcome of the coin toss is Heads?

Elga, A. “Self‐locating belief and the Sleeping Beauty problem”, Analysis 60, 143–147 (2000)

It has become traditional to add that Sleeping Beauty is initially put to sleep on Sunday and is either woken up on Monday (Heads) or Monday and Tuesday (Tails). Then on Wednesday she is woken for the final time and the experiment is over. She knows in advance exactly what is going to take place, believes the experimenters and trusts that the coin is fair.

Much like the Monty Hall problem, Sleeping Beauty has stirred enormous controversy. There are two primary schools of thought on this problem. The thirders and the halfers. Both sides have a broad range of arguments, but put simply they are as follows.

Halfers argue that the answer is 1/2. On Sunday Sleeping Beauty believed that the chance of Heads was 1/2, she has learned nothing new when waking and so the chances are still 1/2.

Thirders argue that the answer is 1/3. If the experiment is repeated over and over again, approximately 1/3 of the time she will wake up after Heads and 2/3 of the time she will wake up after tails.

I first came across this problem myself when a friend alerted me to a blog post by my former supervisor Bob Walters, who describes the thirder position as an “egregious error”. But as Bob notes, there are many in the thirder camp, including Adam Elga himself, physicist Sean Carroll and statistician Jeffrey Rosenthal.

As for my own view, I will leave you in suspense for now, mainly because I’m still thinking it through. Although superficially similar, I believe that it is a far more subtle problem than the Monty Hall problem and poses challenges to what it means to move the pure mathematical theory of probability to a real world setting. Philosophers distinguish between the mathematical concept of “probability” and real world “credence”, a Bayesian style application of probability to real world beliefs. I used to think that this was a bit fanciful on the part of philosophers. Now I am not sure sure: applying probability is harder than it looks.

Let me know what you think!

Image Credit: Serena-Kenobi

Government spending

16 Replies

Before, during and after this month’s budget, Treasurer Joe Hockey sounded dire warnings about Australia’s “budget emergency”. Amidst this fear-mongering, it was a pleasant relief to come across a dissenting view. In a recent interview on 2SER Dr Stephanie Kelton (Department of Economics at the University of Missouri in Kansas City) argued that the government budget is very different from a household budget, however appealing that analogy might be. Governments like the Australian government, with its own free-floating currency can spend more than they take in taxation without worrying about running out of money. While the economy is weak, the government can comfortably run a deficit. The constraint to worry about is the risk of inflation, which means curbing spending once the economy heats up.

I posted a link to Facebook, and immediately drew comment from a more ~~conservatively~~ libertarian-minded friend: “of course a deficit is a bad thing!”. Pressed for an explanation, he argued that government spending was inefficient and “crowded out” more productive private sector investment. This did not surprise me. Deep down, the primary concern of many fiscal conservatives is government spending itself, not a deficit. This is easy to test: ask them whether they would be happy to see the deficit closed by increased taxes rather than decreased spending. The answer is generally no, and helps explain why so many more traditional conservatives are horrified by the prospect of the Coalition’s planned tax on higher income earners….sorry, “deficit levy”.

From there, the debate deteriorated. North Korea was compared to South Korea as evidence of the proposition that government spending was harmful, while a left-leaning supporter asked whether this meant Somalia’s economy should be preferred to Sweden’s. Perhaps foolishly, I proffered a link to an academic paper (on the website of that bastion of left-wing thought, the St.Louis Fed) which presented a theoretical argument to the “crowding out” thesis. My sparring partner then rightly asked whether the thread was simply becoming a rehash of the decades old Keynes vs Hayek feud, a feud best illustrated by Planet Money’s inimitable music video.

Macroeconomic theory was never going to get us anywhere (as I should have known only too well). Instead, the answer lay in the data, with more sensible examples than North Korea and Somalia. Aiming to keep the process fair, avoiding the perils of mining data until I found an answer that suited me, here was my proposal:

I’m going to grab a broad cross-section of countries over a range of years and compare a measure of government expenditure (as % of GDP to be comparable across countries) to a measure of economic success (I’m thinking GDP per capita in constant prices).

If indeed government spending is inherently bad for an economy, we should see a negative correlation: more spending, weaker economy and vice versa. My own expectation was to see no real relationship at all. In a period of economic weakness, I do think that government spending can provide an important stimulus, but I do not think that overall government spending is inherently good or bad.

The chart below illustrates the relationship for 32 countries taken from the IMF’s data eLibrary. To eliminate short-term cyclical effects, government spending and GDP per capita (in US$ converted using purchasing power-parity) was averaged over the period 2002-2012.

The countries in this IMF data set are all relatively wealthy, with stable political structures and institutions. All but one is classified as a “democracy” by the Polity Project (the exception is Singapore, which is classified as an “anocracy” due to an assessment of a high autocracy rating). This helps to eliminate more extreme structural variances between the countries in the study, providing a better test of the impact of government spending. Even so, there are two outliers in this data set. Luxembourg has by far the highest GDP per capita and Mexico quite low GDP per capita, with the lowest rate of government spending.

The chart below removes these outliers. There is no clear pattern to the data. There is no doubt that government spending can be well-directed or wasted, but for me this chart convincingly debunks a simple hypothesis that overall government spending is necessarily bad for the economy.

Now look for the cross (+) on the chart: it is Australia (IMF does not include data for New Zealand and we are the sole representative of Oceania). Despite Hockey’s concerns about a budget emergency, Australia is a wealthy country with a relatively low rate of government spending. Among these 30 countries, only Switzerland and South Korea spend less. These figures are long run averages, so perhaps the “age of entitlement” has pushed up spending in recent years? Hardly. Spending for 2012 was 35.7% compared to the 2002-2012 average of 35.3%. The shift in the balance of government spending from surplus to deficit is the result of declining taxation revenues rather than increased spending. Mining tax anyone?

Randomness revisited (mathsy)

2 Replies

My recent randomness post hinged on people’s expectations of how long a run of heads or tails you can expect to see in a series of coin tosses. In the post, I suggested that people tend to underestimate the length of runs, but what does the ~~fox~~ maths say? The exploration of the numbers in this post draws on the excellent 1991 paper “The Longest Run of Heads” by Mark Schilling, which would be a good starting point for further reading for the mathematically inclined.. When I ran the experiment with the kids, I asked them to try to simulate 100 coin tosses, writing down a sequence of heads and tails. Their longest sequence was 5 heads, but on average, for 100 tosses, the length of the longest run (which can be either heads or tails) is 7. Not surprisingly, this figure increases for a longer sequence of coin tosses. What might be a bit more surprising is how slowly the length of longest run grows. Just to bump up the average length from 7 to 8, the number of tosses has to increase from 100 to 200. It turns out that the average length of the longest run grows approximately logarithmically with the total number of tosses. This formula gives a pretty decent approximation of the expected length:

average length of longest run in n tosses ≃ log₂n + 1/3

The larger the value of n, the better the approximation and once n reaches 20, the error falls below 0.1%.

Growth of the Longest Run

However, averages (or, technically, expected values) like this should be used with caution. While the average length of the longest run seen in 100 coin tosses is 7, that does not mean that the longest run will typically have length 7. The probability distribution of the length of the longest run is quite skewed, as is evident in the chart below. The most likely length for the longest run is 6, but there is always a chance of getting a much longer run (more so than very short runs, which can’t fall below 1) and this pushes up the average length of the longest run.

Distribution of the Longest Run in 100 coin tosses

What the chart also shows is that the chance of the longest run only being 1, 2 or 3 heads or tails long is negligible (less than 0.03%). Even going up to runs of up to 4 heads or tails adds less than 3% to the cumulative probability. So, the probability that the longest run has length at least 5 is a little over 97%. If you ever try the coin toss simulation experiment yourself and you see a supposed simulation which does not have a run of at least 5, it’s a good bet that it was the work of a human rather than random coin. Like the average length of the longest run, this probability distribution shifts (approximately) logarithmically as the number of coin tosses increases. With a sequence of 200 coin tosses, the average length of the longest run is 8, the most likely length for the longest run is 7 and the chances of seeing a run of at least 5 heads or tails in a row is now over 99.9%. If your experimental subjects have the patience, asking them to simulate 200 coin tosses makes for even safer ground for you to prove your randomness detection skills.

Distribution of the Longest Run in 200 coin tosses

What about even longer runs? The chart below shows how the chances of getting runs of a given minimum length increase with the length of the coin toss sequence. As we’ve already seen, the chances of seeing a run of at least 5 gets high very quickly, but you have to work harder to see longer runs. In 100 coin tosses, the probability that the longest run has length at least 8 is a little below 1/3 and is still only just over 1/2 in 200 tosses. Even in a sequence of 200 coin tosses, the chances of seeing at least 10 heads or tails in a row is only 17%.

Longest Run probabilities

Getting back to the results of the experiment I conducted with the kids, the longest run for both the real coin toss sequence and the one created by the children was 5 heads. So, none of the results here could help to distinguish them. Instead, I counted the number of “long” runs. Keeping the distribution of long runs for 100 tosses in mind, I took “long” to be any run of 4 or more heads or tails. To calculate the probability distribution for “long” runs, I used simulation*, generating 100,000 separate samples of a 100 coin toss sequences. The chart below shows the results, giving an empirical estimate of the probability distribution for the number of runs of 4 or more heads or tails in a sequence of 100 coin tosses. The probability of seeing no more than two of these “long” runs is only 2%, while the probability of seeing 5 or more is 81%.

These results provide the ammunition for uncovering the kids’ deceptions. Quoting from the Randomness post:

One of the sheets had three runs of 5 in a row and two runs of 4, while the other had only one run of 5 and one run of 4.

So, one of the sheets was in the 81% bucket and one in the 2% bucket. I guessed that the former was the record of coin tosses and the second was devised by the children. That guess turned out to be correct and my reputation as an omniscient father was preserved! For now.

If you have made it this far, I would encourage you to do the following things (particularly the first one):

Listen to Stochasticity, possibly the best episode of the excellent Radiolab podcast, which features the coin toss challenge
Try the experiment on your own family or friends (looking for at least 3 runs of 5 or more heads or tails and ideally at least one of 6 or more)
Share your results in the comments below.

I look forward to hearing about any results.

* UPDATE: I subsequently did the exactly calculations, which confirmed that these simulated results were quite accurate.

Do Daleks use toilet paper?

6 Replies

I have been watching some (very) old Doctor Who episodes, including the first ever serial featuring the archetypal villains, the Daleks. In this story, the Daleks share a planet with their long-time enemies, the Thal. After a war culminating in the denotation of a neutron bomb, both races experience very different mutations. The Daleks have become shrunken beasts that get about in robotic shells, while the more fortunate Thals mutated into peace-loving blondes.

The Thals hope to make peace with the Daleks, but the Daleks have more fiendish plans and plot to lure the Thals into their city with a gift of food and then ambush them. It is a good plan, but it is the choice of gifts that left me bemused. There is plenty of fruit and some large tins whose contents remain undisclosed. These may be reasonable choices, although I do find it hard to picture the Daleks stacking melons with their plunger hands. But the trap also appears to feature stacks of toilet paper. Granted, toilet paper may be an appealing luxury for the Thal, who have been trekking through the jungle for a year, but the real question here is, why do Daleks even have toilet paper?

Randomness

4 Replies

With three children, I have my own laboratory at home for performing psychological experiments. Before anyone calls social services, there is an ethical committee standing by (their mother).

This evening, I tried out one of my favourites: testing the perception of randomness. Here is the setup: I gave the boys two pieces of paper and a 20 cent coin. I was to leave the room, then they had to decide which of the two sheets of paper would be decided by the boys and which by a coin. Having made their choice, they then had to write down on one of the sheets their best attempt at a “random” sequence of 100 heads (H) and tails (T). Having done that, they were then to toss the coin 100 times, writing down on the other page the sequence of heads and tails that came up. I would then return to the room and guess which one was determined by the toss of the coin, and which by the boys.

I identified which sequence was which in less than 30 seconds. How did I do it?

The trick is to look for the longer sequences. Much like the gambler at the roulette wheel, the kids assume that a run of heads cannot last too long. One of the sheets had three runs of 5 in a row and two runs of 4, while the other had only one run of 5 and one run of 4. I correctly picked that the sheet with more long runs was determined by the coin toss.

Try it yourself sometime. If you see a run of 6 or more (which is in fact quite probable in a sequence of 100 coin tosses), you can quite confidently pick that as the coin toss, unless your subject has been well schooled in probability.

Our intuition struggles with randomness. We tend to assume randomness is more regular than it is. On the other hand, we also try to find patterns where there is only randomness, whether it is the man in the moon, clouds that look like things, the face of Mary on a piece of toast or, perhaps, an explanation for the disappearance of MH 370.

Chinese non-residents…in China

4 Replies

Recently I travelled to China for the first time. My first glimpse of Beijing took in the Escher-like headquarters of Chinese TV station CCTV. It is an extraordinary building and to get a proper sense of it, you have to see it from a number of different angles.

Driving across the city, impressed by the scale of the place, I asked one of my hosts about the population of Beijing. He told me there were about 40 million, including non-residents. Almost double the entire population of Australia. Maybe it’s an exaggeration, but more than the figure itself it was the reference to “non-residents” that piqued my interest. Where there really so many people moving to China as to have a significant impact on the population of the capital?

Later, I learned that these non-residents were in fact people from other provinces. Under China’s Hukou system, restrictions are imposed on people’s ability to move from one part of the country to another. Many people from rural areas are drawn to cities to find work, but without residency rights for the city in which they work they cannot access public education or health care. So, Beijing is full of married men who have left their families at home in the provinces. Living in tiny apartments, they work all year and then travel back to their families for Chinese New Year, taking their earnings with them.

Being used to freedom of movement in Australia, it’s hard not to see this as a harsh system. But, reflecting on the numbers, China is a country of 1.3 billion people; if there are already 30 to 40 million people in Beijing, how would the city cope with a sudden influx of millions more? Only a few days ago, the central committee of China’s communist party released new targets to increase urbanisation from 53.7% of the population to 60% by 2020. This plan involves granting urban hukou status to an additional 100 million rural migrant workers. Even so, another 200 million migrants will remain non-residents. It is sobering to consider the potential consequences of granting full freedom of migration to the entire population rather than managing the process in this highly controlled fashion.

I’m not about to renounce my belief in democracy (however challenged it may be in many Western countries today), but, much like the CCTV building, it seems that to better understand China, you have to see it from a number of different angles.

I’m with Felix

4 Replies

FT blogger Felix Salmon and venture capitalist Ben Horowitz have very different views of the future of Bitcoin. Salmon is a skeptic, while Horowitz is a believer. A couple of weeks ago on Planet Money they agreed to test their differences with a wager.

Rather than a simple bet on the value of Bitcoin, the bet centres of whether or not Bitcoin will move beyond its current status, as a speculative curiosity, to serve as a genuine basis for online transactions. The test for the bet will be a survey of listeners in five years’ time. If 10% or more of listeners are using Bitcoin for transactions, Horowitz wins. If not, Salmon wins. The winner will receive a nice pair of alpaca socks.

I have been fascinated by Bitcoin for some time now and have a very modest holding of 1.6 Bitcoin. Nevertheless, I believe that Felix is on the right side of the bet. I have no doubt that the technological innovation of Bitcoin will inform the future of digital commerce, but Bitcoin itself will not become a mainstream medium of exchange.

Volatility

Only days after the podcast, the price of Bitcoin tumbled as MtGox, the largest Bitcoin exchange in the world, suspended Bitcoin withdrawals due to software security problems. Sadly, this means my own little Bitcoin investment halved in value. It also highlights how much of a roller-coaster ride the Bitcoin price is on. As long as Bitcoin remains this volatile, it cannot become a serious candidate for ecommerce. It is just too risky for both buyers and sellers. Horowitz acknowledges that the Bitcoin market is currently driven by speculators, but is confident that the price will eventually stabilise. I doubt this. Even during its most stable periods, the volatility of Bitcoin prices is far higher than traditional currencies, and has been throughout its five year history.

The Ledger

One of the key innovations of Bitcoin is its distributed ledger. Everyone installing the Bitcoin wallet software ends up downloading a copy of this ledger, which contains a record of every single Bitcoin transaction. Ever. As a result, there is no need for a central authority keeping tabs on who owns which Bitcoin and who has made a payment to whom. Instead, every Bitcoin user serves as a node in a large peer-to-peer network which collectively maintains the integrity of this master transaction ledger. This ledger solves one of the key problems with digital currencies: it ensures that I cannot create money by creating copies of my own Bitcoin. The power of the ledger does come at a cost. It is big! On my computer, the ledger file is now almost 12 gigabytes. For a new Bitcoin user, this means that getting started will be a slow process, and will make a dent in your monthly data usage. A popular way around this problem is to outsource management of the ledger to an online Bitcoin wallet provider, but that leads to the next problem.

Trust Problems

A big part of the appeal of Bitcoin to the more libertarian-minded is that you no longer have to place trust in banks, government or other institutions to participate in online commerce. In theory, at least. If you decide to use an online Bitcoin wallet service to avoid the problem of the large ledger, you have to trust both the integrity and the security capability of the service provider. The hacking of inputs.io shows that this trust may well be misplaced. Even if you have the patience and bandwidth to maintain your own wallet, trust is required when buying or selling Bitcoin for traditional currency. There are many small Bitcoin brokers who will buy and sell Bitcoin, but invariably you have to pay them money before they give you Bitcoin, or give them Bitcoin before you get your money. Perhaps the big exchanges, like MtGox, should be easier to trust because their scale means they have more invested in their reputation. But they are not household names, the way Visa, Mastercard or the major banks are. Growth of commerce on the internet has been built on trust in the names providing the transactions more than trust in the technology, which most people don’t understand. I would be very surprised to see the same level of trust being established in the Bitcoin ecosystem, unless major financial institutions begin to participate.

The Authorities

But will banks jump onto the Bitcoin train? I doubt it. Not because they are afraid of the threat to their oligopoly—most bankers still only have the vaguest idea of exactly what Bitcoin is, or how it works. What they do know is that virtual currencies are attractive to criminals and money launderers. Last year saw the FBI crackdown on Liberty Reserve, followed by the crackdown on the underground black-market site Silk Road. More recently, the CEO of one of the better-known Bitcoin exchanges was arrested for money-laundering. In the years since September 11, the regulatory obligations on banks to ensure they do not facilitate money laundering have grown enormously. The anonymity of Bitcoin makes it hard for banks to “know their customer” if they deal with Bitcoin and as law-enforcement increases its focus on virtual currencies, providing banking services to Bitcoin brokers becomes less appealing for banks. When I bought my Bitcoin last year, I used the Australian broker BitInnovate. For several months now, their Bitcoin buying and selling services have been suspended and, I’m only guessing, this may be because their bank closed down their accounts. To become a widely-accepted basis for commerce, Bitcoin will necessarily have to interface effectively with the traditional financial system. At the moment, the prospects for this don’t look good.

For these reasons, I think Felix has a safe bet, and can look forward to cosy feet in alpaca socks. But, even if Bitcoin does not become widely accepted, its technological innovations may well revolutionise commerce anyway. Banks around the world can adopt ideas like distributed ledgers and cryptographically secure, irrevocable transactions to make the mainstream global payments system more efficient.

Shark season

21 Replies

Summer in Australia comes with cicadas, sunburn and, in the media at least, sharks. So far, I have learned that aerial shark patrols are inefficient (or perhaps not) and that the Western Australian government plans to keep swimmers safe by shooting big sharks.

Sharks are compelling objects of fear, right up there with spiders and snakes in the package of special terrors for visitors to Australia. As good hosts, we are quick to reassure: sharks may be the stuff of nightmares and 70s horror movies, but attacks are rare.

But, exactly how rare is death by shark? Over a Boxing Day lunch, I heard an excellent ‘statistic’, designed to reassure a visiting American. Apparently, more people are killed each year in the US by falling vending machines than are killed by sharks around the world. I was skeptical, but had no data to hand. Later, with the help of Google, I discovered that this statistic is 10 years old and the source? Los Angeles life guards. The tale has, however, become taller over time. Originally, vending machine deaths in the US were compared to shark attack fatalities in the US, not the entire world.

While data on vending machine related deaths are hard to come by, subsequent attempts to validate the story concluded that it was plausible, on the basis that there were two vending machine deaths in 2005 in the US but no fatal shark attacks.

Fun though the vending machine line may be, it is not relevant to Australia and, if you are on the beach contemplating a quick dip, then the risk of a shark attack is certainly higher in the sea than death by vending machine. Local data is in order.

According to the Taronga Zoo Australian Shark Attack File (ASAF):

In the last 50 years, there have been 50 recorded unprovoked fatalities due to shark attack, which averages one per year.

Fatalities have been higher than average over the last couple of years. The ASAF recorded two deaths in 2012 and, although validated figures for 2013 are yet to be published, six deaths have been reported over the last two years, suggesting that fatalities rose further to four this year.

To compare shark fatalities to other causes of mortality, a common scale is useful. My unit of choice is the micromort. A one-in-a-million chance of death corresponds to a micromort of 1.0, a one-in-ten-million chance of death to a micromort of 0.1. Taking the recent average death rate of three per year (more conservative than the longer run average of one), and a population of 23 million in Australia leads to a figure of 0.13 micromorts for the annual risk of death for a randomly chosen Australian.

The most recent data on causes of death published by the Australian Bureau of Statistics (ABS) are for 2009. That year, three people were killed by crocodiles. Sharks are not specifically identified, but any fatal shark attacks would be included among the three deaths due to ‘contact with marine animals’. The chart below illustrates the risk of death associated with a number of ‘external causes’. None of these come close to heart disease, cancer or car accidents. Death by shark ranks well below drowning, even drowning in the bath, as well as below a variety of different types of falls, whether from stairs, cliffs or ladders.

Annual risk of death in Australia (2009 data)*

Of course, you and I are not randomly chosen Australians and our choices change the risks we face. I am far less likely to suffer death by vending machine if I steer clear of the infernal things and I am far less likely to be devoured by a shark if I stay out of the water.

So, care should be taken when interpreting the data in the chart. Drug addicts (or perhaps very serious Hendrix imitators) are far more likely to asphyxiate on their own vomit than summer beach-goers. The fairest point of comparison is drowning in natural waters. At almost 3.5 micromorts, drownings in the sea (or lakes and rivers) is more than 25 times more common than fatal shark attacks. And the risk of both can be reduced by swimming between the flags.

What does that leave us with for conversations with foreign visitors? If you are headed to the beach, the risk of shark attack would be higher than death by vending machine, but it is still very low. The drive there (at 34.3 micromorts) is almost certainly more dangerous.

I will be taking comfort from my own analysis as I am heading to Jervis Bay tomorrow and sharks were sighted there this weekend:

Bendigo Bank Aerial Patrol spotted up to 14 sharks between 50 and 100 metres from shore at various beaches in Jervis Bay. [The] crew estimated the sharks at between 2.5 and 3.5 metres in length at Nelsons, Blenheim, Greenfields, Chinaman’s Beach and Hyams Beaches.

The beaches are un-patrolled, so wish me luck…but I don’t think I’ll need it.

* The figure for ‘Shark attack’ is based on the estimate of three deaths per year rather than the ABS data.

Qantas and Adobe

Security can be tricky

9 Replies

Qantas has recently launched Qantas cash, a pre-paid Mastercard which you can charge up with cash in multiple currencies. The contemporary equivalent of traveller’s cheques, cards like this can be as convenient as a credit card with the added advantage of reducing the uncertainty associated with exchange rate volatility. If you have a rough idea of how much you will need in euro, you can charge up the card with euro at today’s exchange rate without having to worry about the Australian dollar dropping in value while you are half way through your trip.

As a Qantas frequent flyer account holder, I received a Qantas cash card in the mail and it seemed worth investigating. However after activating the card, my interest in the card itself was quickly displaced by disappointment in the insecure design of the Qantas cash website.

Computer security is not easy. It should be left to the experts. I am no expert myself, but I have listened to enough of the Security Now podcast to recognise poor security when I see it.

The first sign of trouble came with setting my password. The password had to be 6 to 8 characters long. A maximum of only 8 characters? The longer the password length, the more secure it is and 8 characters is far too short for a secure password.

Somewhat disconcerted, I pressed on, creating a password made up of 8 random characters. Random passwords are far more secure than real words (or even transparently modified “w0rd5”). They are also impossible to remember, but there are plenty of secure password storage tools (such as LastPass) that make that unnecessary.

Having set everything up, I was then prompted to log in. Unexpectedly, instead of being prompted to enter my password, I was asked to enter the “3rd, 4th and 5th character of the password”. Alarm bells started ringing. Quite apart from the irritation that this caused as it prevented LastPass from automatically filling in the password, it confirmed my initial fears that the website’s security model was flawed.

What I had realised was that Qantas servers must be storing passwords. For anyone unfamiliar with password security, this may seem blindingly obvious. If the servers don’t store the password, how can the website confirm you have entered the correct password when you log in?

In fact, there is a far more secure approach, which makes use of so-called “one way functions“. A one-way function takes a string of characters (a password, for example) as input produces a different string of characters as its output. The key feature of a one-way function is that it extremely difficult to reverse the process: given the output, working out what the input must have been is computationally highly intensive. Applying a one-way function is also known as (cryptographic) “hashing”.

Armed with a good one-way function, instead of storing passwords, a web server can store a hash of the password*. Then, whenever a user enters a password, the web site applies the one-way function and compares the result to its database. The password itself can be discarded immediately. The webserver’s user database should only ever contain hashes of user passwords and never the “plain text” original version of the password.

While this approach to password storage is well-established practice in the security community, many corporate websites are not designed by security experts. Back in 2011, hackers were able to get hold of more than a million passwords from Sony which had been stored in plain text.

Unfortunately, it would appear that Qantas cash is not following best practice in its website security. If the site was only storing hashed passwords, it would be impossible for the site to verify whether users were correctly entering the 3rd, 4th and 5th character of the password. Taking a password hash and trying to determine individual characters of the original password is just as difficult as reverse engineering the whole password.**

I then called Qantas cash to seek clarification. I was assured that all passwords were “encrypted” using the same security techniques that any other commercial website, such as Amazon, would use. Furthermore, the requirement to enter individual characters of the password was an additional security measure to prevent users from copying and pasting passwords.

This did not reassure me. Even if the passwords are encrypted, the Qantas cash server itself clearly has the capability of decrypting the passwords, which makes it just as vulnerable as Sony. I am also sure that Amazon does not use this approach. And preventing copying and pasting is a furphy. By preventing users from using secure password stores, this approach simply encourages the use of weaker passwords.

The Qantas cash developers may think they have come up with some excellent security features. But these developers are clearly not experts in security and, as a result, have produced a far less secure site. The call centre promised that the technical team would email me more details of the site’s security. My hopes are not high.

Needless to say, I will not be using the Qantas cash card. This is an e-commerce site, not a movie chat forum. When money is involved, security should be paramount.

Keep your eyes open for news about a Qantas cash website hack.

* Strictly speaking, a “salted hash” should be stored to add an additional layer of security and protect against the use of rainbow tables.

** In principle, Qantas could store hashes of three character combinations (56 hashes would have to be stored or 336 if order is significant). In practice I doubt this is being done.