Graphing using R

4 Replies

R Project logo Long-time readers of the Stubborn Mule will know that charts are a regular feature here. Almost all of these charts were produced using the R statistical software package which, in my view, produces far superior results to the most commonly used graphing tool: Excel. As a community service to help rid the world of horrible Excel charts, here is a quick tutorial on charting using R. Since R is a powerful and versatile tool, there is a lot more to it than covered here, so there may be more tutorials to come.

Installing and Running R

The first step is to get R installed on your computer. R is open source and can be downloaded for free from the Comprehensive R Archive Network (CRAN). It comes in many flavours: Mac, Windows and Linux.

Once you have installed R and have fired it up, you are presented with something that looks very different to Excel. This is the first indication that R is an interactive programming environment not a spreadsheet. You will see various messages, including copyright information, some instructions on how to display licence information, how to run a demo, get help, and finally you are presented with a command prompt: “>”. R is now waiting for you to type commands.

As an example, try entering the following command:

getwd()

This will display the current “working directory” (hence “wd”), which is the default folder that R will use for reading and writing data. You can easily change the working directory, either by using the drop-down menus (which menu option varies depending on whether you are using Windows, Mac or Linux) or by using the setwd command:

setwd("/Documents/Mule Docs")

Unless you have a “Mule Docs” folder in a “Documents” folder, you will need to substitute the name of one of your own folders, otherwise you will get an error message. Note that you need to use forward slashes (“/”) rather than backslashes (“\”) even on Windows.

You can see detailed explanations of any R command by prefixing the name of the command with a question mark:

?setwd

This is short for help(setwd). Of course, this assumes you know the name of the command already. To search the documentation for a keyword, use a double question-mark. For example

??median

will show a list of all the commands which feature the word “median” in their documentation. This is short for help.search(“median”). Note the use of double quotes (“) here, not required in the ?? syntax.

Reading Data and Charting

To get started, here is a simple data file in CSV fomat (“comma separated values”). Download it and save it in your working directory (or save it somewhere else and then change R’s working directory to where you just saved the file). You can then load the data into R with the following command:

x <- read.csv("demo.csv")

While the read.csv part is self-explanatory, the “<-” may look a little odd. It is the assignment operator. Whereas most programming languages simply use an “=” to assign to variables, R uses what is intended to look like an arrow. In this case, you should interpret the command as saying “read the contents of the file demo.csv and place the result in the variable x“. To see the contents of x, you can simply type x at the command line and press return, which will display a table with all the data read from the demo.csv file. When dealing with larger “data frames” (to use the R lingo for this type of object), having that much data flash by may not be very useful. Some other useful commands for quickly inspecting your data are:

head(x)
tail(x)
summary(x)

Now you are ready for your first graph. Try this command:

plot(x)

You should see a simple, clean scatter-plot. If you would prefer a line graph, this is easily done too.

plot(x, type="l")

The plot function has many options, which you can explore in the documentation (just enter ?plot). There are also various commands for further annotations for your chart. Try the following commands:

grid()
axis(side=4)
text(2, -4, "Random Walk")

These will add gridlines, put axis labels on the right-hand sides (R numbers chart sides from 1 to 4 starting from the bottom and working clockwise) and finally displays text on the chart.

Using Program Files

Using R interactively like this is useful for familiarising yourself with the system and for performing quick calculations, but if you find yourself wanting to make small changes here and there, it will quickly become annoying re-typing long commands. This is when you should move to using program files. All that this involves is saving a series of R commands to a file using a text editor (you can just use a simple text editor like Notepad or TextEdit, but many fancier applications can help out by automatically highlighting R commands in different colours, a trick known as “syntax highlighting”). Here is one I prepared earlier: demo.R (by convention, R files are given the .R extension). You can download this and save it into the same folder as the demo.csv file. To execute a program file once you have saved it, you use the source command:

source("demo.R")

This example will also produce a chart of the demo data, but this time it saves the result to an image file (using the Portable Network Graphics image format). This is done using the png command:

png("demo.png", width=400, height=400)

The main parameters for this command are the filename of the image you want to produce and the size of the image. After you execute all of your desired charting commands, you must close off the graphics “device” and save the results, which is done using the following command:

dev.off()

To find out more about graphics “devices” in R, including saving to other file formats (such as PDF or JPEG), have a look at ?Devices.

So that’s it. You are up and running producing charts with R. To go further from here, while you wait for further tutorials, you can explore some of the R files I have used to produce charts for the blog. I store quite a few of them here on github.

The Australian Resources Tax

8 Replies

The recent announcement by the Australian Treasurer of plans to introduce a “Resource Super Profits Tax” (RSPT) has led to the longest discussion thread on the Mule Stable yet. A lot of the discussion turned on whether or not share investors can be considered to have lost anything when share prices fall if they have not sold their shares.

Whether or not “unrealised losses” should be considered real losses takes us back to an oft-visited topic: the nature of money. Money has many guises: store of wealth, medium of exchange and, most relevant here, unit of value. Finance has its jargon like any other discipline and when money serves as a unit of value, it is known as a numéraire. Today, however, I will not explore the theory of money any further (although, you can trawl through the Mule Stable discussion to gather some of my thoughts). Instead, I will focus on what has happened to mining stocks.

The chart below shows the performance of the S&P/ASX 300 share price and the Metals and Mining index. While not quite as broad as the All Ordinaries index, the Australian stock market is dominated by large companies and in fact the market capitalization* of the ASX 300 is around 85% of the All Ordinaries, so it does give a very good indication of the performance of the overall market. The Metals and Mining index simply consists of those companies in the ASX 300 that are categorised as being (no surprise) in the business of metals or mining. In order to provide a direct comparison, both of these indices have been scaled to a common base of 100 on 30 April. This was this the Friday before the weekend announcement of the RSPT.

Performance of the Mining Sector following the RSPT announcement*

As the chart clearly shows, the metals and mining index certainly did suffer more than the market as a whole in the first couple of days after the announcement. By the end of Tuesday, resources had fallen 4% more than the ASX 300. Since the RSPT can only serve to decrease not increase profits of resources companies, this fall would seem quite reasonable. Curiously though, this week resources closed the gap once more. In fact, the resources sector has now performed 0.35% better than the overall market!

Of course, one could argue that the sector returns would have been even better over the last two weeks if the tax had never been announced. That may well be the case, but it is hard to argue that the Government had caused a terrible mischief to the superannuation savings of all working Australians when resource have, well, matched the performance of the broader market.

*Data source: Standard and Poor’s

A spam attempt gem

3 Replies

I have been getting a few very enjoyable spam attempts on the blog of late. While the filter captures the usual Russian porn dross, from time to time comments slip through the filter and it falls to me to moderate them. This little gem appeared on a two year old post about wandering the streets of Newtown with my (then) five year-old son looking at the annual “Art on the Streets” displays in shop windows.

Gratitude for posting this posting. I’m decidedly frustrated with struggling to researching out pertinent and intelligent commentary on this issue. Everybody today goes to the very far extremes to either drive household their viewpoint that possibly: everyone else in the planet is wrong, or two that everyone but them does not really understand the situation. Many regards for your concise, pertinent insight.

Touched though I may be to have my insights described as concise (rarely) and pertinent (perhaps), this comment is going into the spam bucket. I will not be giving this particular spammer any free traffic to their website.

More on “Five Down”

2 Replies

Yesterday’s puzzle “Five Down” stimulated a fair amount of discussion both in the post’s comments section and via email. I also exchanged emails on the topic with the author of Futility Closet (which is where I came across the puzzle) and he told me that the puzzle generated a lot of correspondence for him too.

All the commenters on the blog came up with the correct solution, but there are quite a few different ways of looking at the problem, all of which help provide insight into the nature of money. Since that is a common topic for this blog, I will consider some of these perspectives here.

First, the solution itself. The question asked was “What was lost in the whole transaction, and by whom?”. Taking the “whole transaction” to include the banker finding the counterfeit note in the first place, the answer is that no-one lost anything, subject to a couple of assumptions. These assumptions are that the banker actually owns the bank and so the bank’s gains or losses are his gains or losses (otherwise we would have to conclude that the banker was up $5 and the bank was down $5), and that the banker and his wife pool their finances (so we treat her debt with the butcher as his debt).

The first way to think of the problem is a variation of the comment from James. Imagine that the $5 was not counterfeit at all and all the same transactions took place with a genuine note. But then imagine that when the banker closed the bank at the end of the day, taking notes and coins back to his safe, the $5 slips from his hands and is blown into the fireplace. There it is quickly consumed by the fire. Earlier in the day, the banker had a windfall of $5, but then he lost the same amount to the fire. He gained in the morning, lost in the evening and, although perhaps disappointed to have lost the $5 again, he was even on the whole transaction. No-one else involved lost either as they had simply performed legitimate transactions, clearing various debts, using a valid $5 note. The question now is, how is anyone any better or worse off in this scenario than if the note had been counterfeit all along? The answer is, they are not.

Now that approach gives the right overall answer, but it may be unsatisfying to some as it doesn’t take account of the fact that a whole series of “invalid” transactions took place with the counterfeit note. This too can be clarified. If the note had been real, then the banker made a gain when he found the note, but finding a counterfeit note involves no gain, because it is worthless. In that case, the gain for the banker comes when he is able to discharge his debt with the butcher using a worthless note. So, he is still ahead early in the day, but the timing is slightly different. With a real note, the gain is in the finding and the transaction with the butcher is a neutral fair trade (legitimate $5 in exchange for a discharged debt). With a counterfeit note, the gain is delayed to the next step in the sequence. Of course, in receiving the counterfeit note, the butcher makes a loss. But then the butcher makes a gain when he in turn is able to discharge his debt to the farmer with a worthless note. And so on. Each person in the chain loses when they receive the $5 but has an offsetting gain when they use it to settle a debt, leaving them whole on the transaction. The chain continues all the way back to the bank, which loses $5 when the laundry woman settles her debt with the dodgy note. Assuming, as we are, that the bank’s loss is the banker’s loss, this simply offsets the gain the banker had when first paying the butcher. Again, everyone comes out even. Of course, if someone other than the banker had been left with the note, they would have been down $5 and the banker up $5. Having the transactions complete a full circle is a key part of the puzzle.

The final perspective is a more technical one. At the heart of money is the notion of a debt. Money is essentially a more convenient way of managing debts. If I buy a cow from a farmer and sell a meat pie to a patron at my restaurant, we could simply agree to record various debts: I owe the farmer one cow, the diner owes me one cow. Of course, this is inconvenient (not to mention risky) as we all have to maintain records denominated in a whole range of different commodities and I don’t really want to discharge my debt to the farmer by giving him a cow back. He has plenty already. Nevertheless, this points to the origins of money. In the excellent (if lengthy) treatise “What is Money” is it observed that “for many centuries, how many we do not know, the principal instrument of commerce was neither the coin nor the private token, but the tally”. Indeed in the Five Down puzzle, there are a whole string of tallies. Each of the players in the story has kept track of a debt due to them and one they owe to another. If the merchant did not owe the laundry woman but instead owed $5 to the farmer, the merchant and the farmer could simply agree to cancel their debts to one another. It is not so easy when the debts extend in a longer chain. Nevertheless, if one were to assemble all the parties in a single room and ask them all to consider their respective debts discharged, they should all readily agree. After all, they all owe $5 and all are owed $5 and it is much easier for everyone if that effective net zero position could really be zero without the fuss of worrying about chasing debts. It would be different if someone was owed more (or less) than they owed. We might call this simultaneous discharging of all the debts “multi-lateral debt netting”. In theory it is very attractive, but in practice we cannot get everyone in the same room to get it done. Effectively, the counterfeit note serves the purpose of facilitating multi-lateral debt netting. Because everything nets out evenly in the story, the counterfeit note can achieve the netting just as effectively as real money. The extra feature real money offers is that if the netting does not quite even out, those owed more than they owe can hang on to the money and use it for netting again in the future. Not so with the counterfeit money: once it is discovered, it loses its power to work. The solution to the puzzle lies in the fact that no debts were left over.

I will end this discussion by reprinting a very similar story that one of my email correspondents sent to me (as I understand it, it is not new but has been updated to fit the times).

It’s a slow day in a dusty little Australian town. The sun is beating down and the streets are deserted. Times are tough, everybody is in debt, and everybody lives on credit.

On this particular day, a rich tourist from down south is driving through town , stops at the local motel and lays a $100 bill on the desk saying he wants to inspect the rooms upstairs in order to pick one to spend the night in.

He gives him keys to a few rooms and as soon as the man walks upstairs, the owner grabs the $100 bill and runs next door to pay his debt to the butcher.

The butcher takes the $100 and runs down the street to repay his debt to the pig farmer.

The pig farmer takes the $100 and heads off to pay his bill at the supplier of feed and fuel.

The guy at the Farmer’s Co-op takes the $100 and runs to pay his drinks bill at the local pub.

The publican slips the money along to the local prostitute drinking at the bar , who has also been facing hard times and has had to offer him “services” on credit.

The hooker rushes to the motel and pays off her room bill to the motel owner with the $100.

The motel proprietor then places the $100 back on the counter so the rich traveller will not suspect anything.

At that moment the traveller comes down the stairs, picks up the $100 bill, states that the rooms are not satisfactory, pockets the money, and leaves town.

No one produced anything. No one earned anything.

However, the whole town is now out of debt and looking to the future with a lot more optimism.

And that, ladies and gentlemen, is how the Australian Government’s stimulus package works!!!

Five Down

16 Replies

One of my favourite blogs is Futility Closet, which is sadly appropriate given its tagline “An idler’s miscellany”. This week it featured a puzzle called Five Down devised by the English mathematician Henry Dudeney. Since the subject of the puzzle is money, it seems like an appropriate one to share here on the Mule.

A banker in a country town was walking down the street when he saw a five-dollar bill on the curb. He picked it up, noted the number, and went to his home for luncheon. His wife said that the butcher had sent in his bill for five dollars, and, as the only money he had was the bill he had found, he gave it to her, and she paid the butcher. The butcher paid it to a farmer in buying a calf, the farmer paid it to a merchant who in turn paid it to a laundry woman, and she, remembering that she owed the bank five dollars, went there and paid the debt.

The banker recognized the bill as the one he had found, and by that time it had paid twenty-five dollars worth of debts. On careful examination he discovered that the bill was counterfeit. What was lost in the whole transaction, and by whom?

I will not reveal the solutiuon here to give you a chance to think about the puzzle. What I will reveal is that the “solution”, originally published in The Strand in 1917, was re-published on the blog yesterday but it is in fact incorrect! Understanding what is wrong with the original solution (and the blog’s author was quick to provide an update following feedback from his readers) gives some insight into two of the roles money plays: a medium of exchange and a store of value.

The Re-birth of the Tablet

6 Replies

Last year I bought a Kindle e-book reader and wrote about its strengths and weaknesses. With the release of Apple’s iPad last month, people keep asking me whether I wish I had waited for that instead. The short answer is, no, but now a fellow gadget-aficionado, Tony (aka @thewordpressguy) has drawn my attention to another new device, the TEGA tablet, and has asked me for an opinion*. I am yet to get my hands on either device, but I won’t let that stop me expressing a view!

The history of tablet computers is showing every sign of repeating the pattern of mp3 players. The first mp3 player pre-dated the iPod by about 5 years, but it was not until Apple entered the market that they really began to take off. Two key factors behind Apple’s success were design and the iTunes store. Having owned a Creative Nomad Jukebox and an iRiver H340 before getting an iPod, I experienced first-hand how much better the user-interface of the iPod was than everything that came before (even if it lagged at times in its technical specifications). The iTunes store was even more important. While early-adopting enthusiasts like me may have had the motivation and patience to convert all of my CDs to mp3 format, this would not be true of most people. The iTunes store provided a simple and reasonably-priced way for people to get content onto their iPods. The rest is history.

Just like iPods, tablets have been around for some time before Apple entered the fray. The term was popularised by Microsoft in 2001, but tablets were around in one form or another well before that (you could even include Apple’s less than successful Newton). Just like the iPod, the iPad is no doubt a superbly-designed piece of hardware, with an intelligent user-interface (I am extrapolating from my experience with the iPhone as well as taking into account the plethora of articles I have read about the iPad). Combine that with the App store, which is to the iPad and iPhone what the iTunes store was to the iPod, and the success of the iPad looks assured.

While the iPod came to dominate the mp3 player market, the iPad may stimulate the emergence of a broader range of alternative tablets, much as the Google Android phone is showing signs of being a serious alternative to the iPhone. Based on Tony’s assessment of the TEGA, it could well be an early example of this phenomenon. While it does not offer Apple design, it does have a few other things instead, such as USB, card-reader ports (rather than having to rely on external adapters as the iPad does) and a built in 3G modem which allows you to pop in your own sim card (so no more 3G dongles). What may hold even greater appeal for some is that it is operating-system agnostic: while most people would buy it with Windows 7 installed, it will also ship with Linux.

The release of the TEGA is an interesting development and I am sure there will be more iPad alternatives to come, but that brings me back to the original question. Do I have Kindle-regret? Would an iPad, a TEGA or something else be better?

I do not. The Kindle certainly has its shortcomings, some of which I discussed in my original review, but here’s what it gives me that the alternatives do not:

electronic-ink display – while the page-turning flicker may be annoying to some, I continue to find it a very easy medium to read, particularly in bright light
battery life – with 3G turned off (except for when I am making a purchase), I get two weeks or more between charges. The iPad offers an impressive 10 hours, so the gap is closing, but the Kindle retains the lead for now. Clocking in at only 2.5 hours, the TEGA remains a laggard on this score and it cannot be a serious contender until this improves.
price – compared to prices of portable computers only a few years ago, with prices starting at US$499 the iPad looks cheap. But at US$259, the Kindle is a lot cheaper. Of course, it cannot do what the iPad can do, but if you are after an e-book reader (as I was) that may not matter. The TEGA is closer to US$1,000 (A$1,187.98) and at that price looks expensive.
continuous partial attention – on a computer I cannot help flicking from email to twitter to following links, so perhaps I suffer from a touch of CPA. What this means is that a single-purpose device like the Kindle is ideal for me and offers a better, less-interrupted reading experience. It may seem absurd to some to want to impose restrictions on a device, but in this case it is an advantage for me.

As a bit of a gadget-obsessive, I may well succumb to the lure of an iPad one day (perhaps 2.0), or indeed a descendant of the TEGA or something similar. For now though, I will happily continue reading on the Kindle.

* In the interest of full disclosure, I should point out that if enough fellow-bloggers post on the topic of the TEGA, Tony will have the option of purchasing a heavily-discounted unit.

Cash rates and mortgage rates

10 Replies

At 2.30pm this afternoon, the Reserve Bank of Australia (RBA) will announce whether or not they will be changing the official cash rate. The bank began increasing the cash rate just over a year ago and since then it has risen by 1.25% to its current level of 4.25%. Most observers in the financial markets expect the RBA to lift the rate another 0.25% today, pointing to last week’s inflation figures as one of the key factors. One of the RBA’s stated objectives it to attempt to maintain “an inflation rate of 2–3 per cent, on average, over the cycle”. That phrase “over the cycle” gives the bank a fair amount of wriggle room, nevertheless it will certainly be concerned that the inflation rate for the March 2010 quarter was 0.9% (or 3.6% on an annualized basis). The RBA uses a number of smoother measures of inflation that aim to strip out some of the volatility in the headline inflation numbers. One of these, the “trimmed mean”, is widely considered to be one of the bank’s favourite smoothed measures and the March quarter reading has now nudged outside the 2-3% target band.

Australian Inflation (2003-2010)

However, the point of this post is not to speculate on the likelihood of another rate hike, but to respond to a query from a reader about the link between the RBA official cash rate and mortgage rates.

In Australia, most home owners have “variable rate” mortgages, so their interest rates can go up and down over the life of the mortgage. It is possible to lock in “fixed rates”, although typically the fixed rate periods only last from between 1 to 5 years. In contrast, the most common mortgage product in the United States is a 30 year fixed rate mortgage, which certainly takes away a significant element of risk for borrowers. An indication of just how significant this risk can be is evident in the fact that the US “sub-prime” mortgages at the heart of the global financial crisis were not fixed rate mortgages and big increases in interest rates tipped many borrowers into default.

The United States aside, variable rate mortgages are very common around the world. What is rather unique to Australia is the way in which rates can vary. In most countries, variable rates are set relative to a standard published benchmark. Since the point of a variable rate is to allow lenders to increase (or decrease) what they charge borrowers as their own short-term borrowing costs change, these benchmarks are typically short-term market rates such as surveyed inter-bank lending rates (typically a 1 month or 3 month rate) or perhaps the official central bank cash rate. These central bank rates apply to very short-term borrowings, typically applying for somewhere between one day and one week (for example, the RBA cash rate is an overnight rate), but have the advantage of not moving around too often. A mortgage-lender might then quote their mortgage rate as, say, 2% over the benchmark rate.

In Australia, things are a little different. The standard variable rate is a so-called “discretionary variable rate”. Put simply, this means that the rate can be whatever the lender wants it to be. Try explaining that to anyone from another country and they would be horrified (I can speak from experience here), but in Australia we seem to have become accustomed to giving all this discretion to our lenders. The usual justification for this approach is that competitive pressure would stop a bank abusing the power inherent in that word “discretionary” and in practice all the banks move their rates in line with the RBA cash rate anyway.

Of course, since the onset of the global financial crisis put pressure on funding costs for banks, this link between the official cash rate and mortgage rates has broken down. Back when the RBA was cutting rates, banks were cutting mortgage rates by less than the RBA cut and, on the way back up, they have been hiking in bigger increments than the RBA*. Some have even increased rates “out of cycle” (i.e. independently of the RBA cash rate movements). Once upon a time, the then Treasurer Peter Costello said that any bank not passing on an RBA cut in full was a “bastard”. Things are different now and if the RBA does indeed increase rates today and you have a mortgage, do not be too surprised if your mortgage rate goes up by even more. If you don’t have a mortgage, try not to gloat. It is unseemly.

* UPDATE: as noted in the comments below, the RBA has estimated that mortgage rates have increased by around 1.3-1.4% more than the official cash rate.

Gigabang for your buck

9 Replies

This week Fairfax reported on Australia’s broadband pricing “war” in an article appearing in both the Sydney Morning Herald and the Age. The publisher thoughtfully spared online readers the egregious chart that it foisted on readers of the paper editions. Judging from this junk (to use the official adjective for low-quality charts), these newspapers should stick to journalism and steer clear of graphics.

The chart in question was brought to my attention by Mule Stable regular @zebra, who also kindly scanned it (and devised the headline of this post), allowing me to reproduce it here. It shows the pricing of a number of broadband internet plans offered by the four largest internet service providers (ISPs) in Australia.

Chart from print edition of The Age (29 April 2010)

It is a busy chart, made difficult to read by a number of ill-advised design decisions:

the horizontal axis reads from right to left rather than the conventional left to right
although labeled “Price vs Download”, price is on the horizontal axis, again violating convention*
repeating the ISP label for every point adds unnecessarily to the busy-ness of the chart and it also makes the legend redundant
labeling each point with the download limit (although not the price), adds more unnecessary ink

These conventions are arbitrary: we could just as well have developed a tradition in the West of reading from right to left, for example. But once a convention is in place, you have to have a very good reason to break with it. Otherwise, you end up making your chart harder for readers to interpret for no good reason.

But perhaps the biggest weakness in the chart is the labeling of the ISPs. Each has its own colour, but this is not enough for the eye to naturally group them together, which makes it hard to track the pricing trend provider by provider. This is easily addressed by connecting the points for each ISP with lines. Once this is done and the other short-comings are also addressed, a couple of anomalies in the data leap out immediately. Compared to their other plans, the Optus 100GB plan and the TPG 150GB appear dramatically over-priced, costing more than other plans that offer more data.

Improved version: Price vs Download limit

Of course, this phenomenon was there in the original chart, but it was hidden. So much so, that the journalist does not appear to have noticed at all as it went unremarked in the article. This is a good example of the power of good charting technique.

There are a number of possible explanations for the anomalous data points. They could simply be errors, although it is certainly not impossible (or perhaps even unlikely) that ISPs have illogical pricing policies. A more likely explanation is that the data includes apples and oranges: the higher-priced plans may be bundles offering additional services such as VOIP that are not included in the other more basic plans. Perhaps if Fairfax had done a better job on the chart in the first place, the journalist may have been prompted to answer this question for us.

* Typically “dependent variables” (the y of “y versus x”) appear on the vertical axis and “independent variables” on the horizontal axis.

The Mule trips up

15 Replies

In my last post, I fell into a common trap when dealing with financial time-series data: I did not adjust for inflation. The post examined recent trends in US personal consumption and concluded with the following chart showing a long history of year-on-year consumption growth.

Year-on-year Growth of US Personal Consumption (1959-2010)

What stands out in the chart is the high rate of growth in the 1970s and 80s, a phenomenon that was picked up in comments on the blog post. Of course, the problem is that inflation was high in the 1970s and 80s and so at least some of that growth can be attributed to rising prices rather than increased consumption of “stuff”.

What I should have done is adjust the personal consumption expenditure (PCE) data for the effect of inflation. This is made easier by the fact that the Bureau of Economic Analysis publishes a companion to the PCE which serves exactly that purpose. The PCE price index (PCEPI) provides a measure of inflation very much like the consumer price index (CPI), but it is based on the particular basket of goods used in the PCE index. Using this index to scale consumption to 2010-equivalent dollars and then looking at the annual growth in this measure of “real” consumption results in a rather different picture.

Year-on-year Growth of US Personal Consumption (1959-2010)

As is often the case with inflation-adjusted data, this chart is noisier than the previous one, and real consumption exhibits bigger swings than the original “nominal” consumption figures. While there is still a declining trend in consumption over time, it is a more modest decline and the 1970s and 80s no longer appear to be a particularly unusual period. Futhermore, the contraction of consumption seen in the wake of the recent economic crisis no longer stands out so dramatically. The falls in real consumption in 1974, 1980 and 1991 were all of a similar size. In fact, the biggest fall was in the 12 months to November 1974. (Note that the github code repository has been updated to include this new chart).

I can be quick to criticize the charts in other publications, so it is only fair that I correct my own mistakes too.

UPDATE: a regular reader has suggested that for a series like the PCE, looking at the original series in nominal (not inflation-adjusted) terms actually is the most appropriate way to look at the data, so that the original post was actually fine. I’m still thinking this through….stay tuned, but it sounds like I will have to correct the correction!

Has the US consumer shaken off the financial crisis?

14 Replies

A few years into the global financial crisis, US unemployment remains high and the economy still appears fragile. Nevertheless, American consumers appear to be returning to their old ways. For years they were seen as the engine of global growth. Their consumption drove exports in countries around the world. However, in the aftermath of the financial crisis, the unemployment rate in the United States soared to double figures, the collapse of property prices around the country eroded the wealth of many Americans and banks reined in their lending, while many borrowers cut their spending to pay down their debts. This appeared to set the scene for a change in the long-standing tradition of US consumer-led economic growth.

However, the latest personal consumption expenditure figures released a few weeks ago by the Bureau of Economic Analysis show another month of consumption growth in February. This is the fifth month in a row of strong personal consumption in the United States. Seen over the broad sweep of the last 50 years, the global financial crisis starts to look like a mere blip in an inexorable climb in personal consumption. (Note: the chart below uses a logarithmic scale so that a straight line indicates a steady rate of growth).

US Personal Consumption Expenditure (1959-2010)

Focusing on the last five years reveals that, while consumption collapsed in mid-2008, by the end of that year consumption began to recover. A little shakily at first, consumers appear to have returned to the pattern seen before the crisis and by late 2009, total personal consumption had exceeded pre-crisis levels.

US Personal Consumption Expenditure (2005-2010)

Looking at year-on-year consumption growth gives further insight into the underlying pattern. Growth peaked in the late 1970s, followed by a slowly declining smoothed trend* to growth rates just above 5% per annum. While the growth in consumption over the 12 months to February 2010 has not quite returned to the 5% level, that period includes weaker figures from early 2009. Annualizing quarterly growth over recent months gives figures back around the 5% mark.

Year-on-year Growth of US Personal Consumption (1959-2010)

This may simply be a bounce back to earlier consumption levels and growth may now slow once more. But, economists, policy-makers and America’s trade partners will all be watching closely to see whether indeed the US consumer has shrugged off the financial crisis and is set to recover its place as the driver of the world economy. This scenario seems all the more likely if consumers forget the lessons offered by the crisis about the perils of excessive debt and once again turn to borrowing to finance consumption.

* For the technically-minded, the smoothing is performed with a LOWESS local regression. The code used to produce all of the charts is available on github.