For some time now, I have been meaning to have a look at Beautiful Soup, a python library designed to make it easy to scrape data from web-sites. Now that I have finally tried it out, I wish I had got to it sooner. It really is very handy and easy to use.
As my first Soup project, I turned to an old Mule favourite: petrol data. The Australian Institute of Petroleum (AIP) publishes retail petrol price data, which it sources in turn from MotorMouth. The price data is spread across individual pages for each state, like this one for Victoria and there are separate pages for unleaded and diesel. It would be nice to pull together all of the data and, since the pages are all laid out in exactly the same way and there is a straightforward naming convention for the urls of each page, this is very easy to do using Beautiful Soup. You can see the results showing average weekly petrol prices for the week ending 18 December in the table below.
With all the data in hand, the obvious questions to ask are: where is the cheapest petrol and where is the most expensive petrol? As you can see in the chart below, Adelaide came in as the cheapest place to fill your tank in late December at 134.9 cent/L, while Broome was the most expensive at 165.9, an impressive 23% mark up over Adelaide.
Top 10 and bottom 10 unleaded petrol prices
(average for the week ending 18 December 2011)
Of course this only gives a snapshot at a point in time: Adelaide may not always offer such good value for money and Broome residents may not always pay such a premium. Unfortunately, I have not been able to find any historical data by town on the AIP website. So I have set my data-scraping routine up to collect the data each week. Some time late this year I will revisit this data to see if any patterns emerge over time.
| State | Town | Weekly Average |
|---|---|---|
| NSW | Albury | 135.5 |
| NSW | Armidale | 143.6 |
| NSW | Ballina | 146.9 |
| NSW | Batemans Bay | 144.3 |
| NSW | Bathurst | 145.9 |
| NSW | Bega | 150.1 |
| NSW | Broken Hill | 145.9 |
| NSW | Bulahdelah | 138.1 |
| NSW | Buronga | 146 |
| NSW | Canberra | 146.4 |
| NSW | Casino | 144.8 |
| NSW | Central Coast | 143.2 |
| NSW | Coffs Harbour | 147 |
| NSW | Cooma | 150.3 |
| NSW | Cootamundra | 149.2 |
| NSW | Deniliquin | 148.9 |
| NSW | Dubbo | 143.1 |
| NSW | Forbes | 147.5 |
| NSW | Forster | 146.9 |
| NSW | Glen Innes | 141.9 |
| NSW | Goulburn | 141.9 |
| NSW | Grafton | 146.3 |
| NSW | Griffith | 144.4 |
| NSW | Gundagai | 141.7 |
| NSW | Gunnedah | 144 |
| NSW | Hay | 145.8 |
| NSW | Inverell | 146.9 |
| NSW | Kempsey | 146.2 |
| NSW | Leeton | 145.5 |
| NSW | Lismore | 144.5 |
| NSW | Lithgow | 139.9 |
| NSW | Mittagong | 149.9 |
| NSW | Moama | 145.9 |
| NSW | Moree | 148.8 |
| NSW | Moruya | 147.6 |
| NSW | Moss Vale | 142.1 |
| NSW | Mudgee | 150.4 |
| NSW | Murwillumbah | 143.1 |
| NSW | Muswellbrook | 146.6 |
| NSW | Newcastle | 144.3 |
| NSW | Nowra | 144.8 |
| NSW | Orange | 147.4 |
| NSW | Parkes | 146.7 |
| NSW | Port Macquarie | 146.5 |
| NSW | Queanbeyan | 146.6 |
| NSW | Singleton | 143.8 |
| NSW | Sydney | 138.2 |
| NSW | Tamworth | 145.7 |
| NSW | Taree | 144.9 |
| NSW | Temora | 148 |
| NSW | Tumut | 145.9 |
| NSW | Tweed Heads South | 141 |
| NSW | Ulladulla | 143.9 |
| NSW | Wagga Wagga | 144.9 |
| NSW | Wauchope | 143.6 |
| NSW | West Wyalong | 149.9 |
| NSW | Wollongong | 143 |
| NSW | Woolgoolga | 145.7 |
| NSW | Yass | 148.4 |
| NT | Alice Springs | 163.4 |
| NT | Darwin | 151.8 |
| NT | Katherine | 146.3 |
| NT | Tennant Creek | 164 |
| QLD | Atherton | 145.3 |
| QLD | Ayr | 145.6 |
| QLD | Biloela | 148.6 |
| QLD | Blackall | 157.8 |
| QLD | Blackwater | 148.2 |
| QLD | Bowen | 146.2 |
| QLD | Brisbane | 141.3 |
| QLD | Bundaberg | 144 |
| QLD | Cairns | 146 |
| QLD | Charters Towers | 149.4 |
| QLD | Childers | 142.9 |
| QLD | Dalby | 142.4 |
| QLD | Emerald | 146.9 |
| QLD | Gladstone | 143.5 |
| QLD | Gold Coast | 141.3 |
| QLD | Goondiwindi | 148 |
| QLD | Gympie | 143.3 |
| QLD | Hervey Bay | 143.9 |
| QLD | Ingham | 142.9 |
| QLD | Innisfail | 145.3 |
| QLD | Kingaroy | 143.7 |
| QLD | Longreach | 153.9 |
| QLD | Mackay | 141.2 |
| QLD | Mareeba | 145.9 |
| QLD | Maryborough | 143.2 |
| QLD | Miles | 151.3 |
| QLD | Moranbah | 146.5 |
| QLD | Mt Isa | 150.4 |
| QLD | Rockhampton | 148.8 |
| QLD | Roma | 148.9 |
| QLD | Sunshine Coast | 141 |
| QLD | Toowoomba | 139.6 |
| QLD | Townsville | 141.9 |
| QLD | Tully | 148.9 |
| QLD | Warwick | 143.7 |
| QLD | Whitsunday | 138.5 |
| QLD | Yeppoon | 148.6 |
| SA | Adelaide | 134.9 |
| SA | Bordertown | 147.9 |
| SA | Ceduna | 150.6 |
| SA | Clare | 138.2 |
| SA | Coober Pedy | 160.4 |
| SA | Kadina | 139.4 |
| SA | Keith | 145.9 |
| SA | Loxton | 147.8 |
| SA | Mt Gambier | 146.2 |
| SA | Murray Bridge | 140.6 |
| SA | Naracoorte | 143.9 |
| SA | Port Augusta | 138.4 |
| SA | Port Lincoln | 145.2 |
| SA | Port Pirie | 139.1 |
| SA | Renmark | 142 |
| SA | Tailem Bend | 146.4 |
| SA | Victor Harbour | 142.9 |
| SA | Whyalla | 143.5 |
| TAS | Burnie | 150.3 |
| TAS | Devonport | 149.8 |
| TAS | Hobart | 150.2 |
| TAS | Huonville | 149.9 |
| TAS | Launceston | 149.9 |
| TAS | New Norfolk | 149.9 |
| TAS | Smithton | 149.5 |
| TAS | Sorell | 144.9 |
| TAS | Ulverstone | 149.9 |
| TAS | Wynard | 152.5 |
| VIC | Ararat | 143.1 |
| VIC | Bairnsdale | 139.6 |
| VIC | Ballarat | 144.4 |
| VIC | Benalla | 145.9 |
| VIC | Bendigo | 142.4 |
| VIC | Cobram | 142.5 |
| VIC | Colac | 145.9 |
| VIC | Corryong | 148.2 |
| VIC | Echuca | 145.9 |
| VIC | Euroa | 140.2 |
| VIC | Geelong | 136.3 |
| VIC | Hamilton | 146 |
| VIC | Horsham | 145 |
| VIC | Koo Wee Rup | 139.5 |
| VIC | Kyabram | 143.9 |
| VIC | Leongatha | 142.9 |
| VIC | Melbourne | 137.9 |
| VIC | Mildura | 147.5 |
| VIC | Moe | 139.9 |
| VIC | Morwell | 142.9 |
| VIC | Portland | 146.9 |
| VIC | Sale | 140.9 |
| VIC | Seymour | 137.9 |
| VIC | Shepparton | 143.6 |
| VIC | Swan Hill | 146.7 |
| VIC | Traralgon | 142.2 |
| VIC | Wallan | 138.1 |
| VIC | Wangaratta | 142.3 |
| VIC | Warrnambool | 143.8 |
| VIC | Wodonga | 139.1 |
| VIC | Yarrawonga | 149.9 |
| WA | Albany | 146.8 |
| WA | Boulder | 151.7 |
| WA | Bridgetown | 144.9 |
| WA | Broome | 165.9 |
| WA | Bunbury | 137.1 |
| WA | Busselton | 141.3 |
| WA | Carnarvon | 154.3 |
| WA | Dongara | 153.9 |
| WA | Esperance | 145.9 |
| WA | Geraldton | 149 |
| WA | Kalgoorlie | 149.6 |
| WA | Karratha | 159.9 |
| WA | Majimup | 143.9 |
| WA | Mount Barker | 149.2 |
| WA | Perth | 138.3 |
| WA | Port Hedland | 160.2 |
| WA | Waroona | 144.4 |
Possibly Related Posts (automatically generated):
- Sydney Petrol Prices (29 June 2008)
- Australian Prices Heading South (29 January 2009)
- Update on Sydney Petrol Prices (21 July 2008)
- Digging into GroceryCHOICE (15 August 2008)
Happy New Year, Stubborn!
Lately, you don’t post often, but when you do, you post very interesting stuff.
This is interesting on two accounts: the Soup and the data itself.
Unfortunately, I’m way too old to get into Python (or any other snake, for that matter :), but the petrol info is interesting.
Say, this might be a wild guess, but I’d guess in Tasmania petrol is cheaper than average. Maybe even cheapest of all.
Switzerland is mostly expensive, but petrol is cheaper than in France or Germany. So the borders have supermarkets on one side and petrol stations on the other.
In contrast, Turkey has the most expensive petrol of any country I have visited. This means that almost everyone uses the buses — and some of the inter-city services run to Swiss precision timing and cleanliness, without being too expensive. I think this – whacking exorbitant taxes on petrol – is a vastly better approach than that taken by China or India, where fuel is actually subsidised!
@Magpie: Happy New Year to you too! 2011 has been a very busy year for me…here’s hoping that 2012 will see fit to allow me to post a little more frequently.
As for Tasmania, for the week ending 18 December, the prices ranged from 144.9 cents/L in Sorrell to 155.5 in Hobart, so in the middle of the spectrum.
@Danny: there’s no doubt that economic incentives can make a big difference.