Where’s your data from? How to take advantage of open data feeds

What challenges have you faced when making transport apps?

mobile friendly websitesmobile friendly websites

mobile friendly websites

What challenges have you faced when making transport apps?

How do you make a high quality transport application when the public data you need comes from a plethora of different sources and the aggregating services are prohibitively expensive? Ekreative has made a range of different transport apps, ranging from uber like taxi services to an eccentric, cocktail delivery service, but nothing has put our team to the test quite like the challenge of creating a comprehensive nation wide bus travel planner for the UK.

Ever popular app type

Making apps that take advantage of public sector information is nothing new, transport apps in particular have been around for a long time now. The widespread availability of that public transport information ensures that apps which use it are plentiful. For aspiring app designers, improving the travel experience of their users seems like an obviously useful product and using freely available data means that content remains relevant without incurring costs. This is reflected in the wide range of transport apps available on the market today, seemingly solving every variety of problem for just about every city and variety of transport.

Who’s not using their open data?

Well, not quite. European PSI Platform publishes a score board showing the extent to which different EU countries take advantage of available public information. Their figures suggest that whilst some countries (especially the UK and Spain), are getting a lot of value out of their open data, many European countries are still missing out on this opportunity to align the public interest with the interests of commercial app developers. Expect this apparent niche in the market to continue to be filled and travel apps to remain a popular development choice for many years to come.

Where does the data come from?

One challenge faced by developers though is the wide range of different sources and types of data being made available. Of course it’s great that there’s so much data out there, it holds a lot of potential value. When making an application that uses just one data source, things are relatively straight forward. London based transport applications for example are able to take advantage of the API’s, data feeds and static files published by Transport For London (TfL) and having all that information collected in one place makes it perfect in terms of convenience for developers. For apps with a wider scope though, the variety of sources can become an issue.

London travel apps are a competitive market

Popular London travel app busguru was the first of its kind when it hit the scene back in 2011. Offering all London bus times and a “my nearest stop” feature as well as access to TfL’s journey planner, the product quickly gained a large following. Since then though, competition has grown from all sides and so to make sure they stay ahead of the game busguru asked ekreative to work with them in developing a UK wide version of their product, covering bus stops not just in London but around the entire country!

mobile friendly websites

mobile friendly websites

mobile friendly websites

Data stream amalgamators can help simplify the process

The idea’s inspired and the data is out there to be used, but it’s published by lots of different groups, in lots of different forms. The problem comes then when you try to pull together such a wide variety of different data sources. Thankfully, we’re not the only people to have encountered this problem and transportapi.com have created an amalgamating service to help developers access a wide range of UK public transport data. It seems ideal; yes it’s a paid service, but with payment tied directly to the number of server calls made by the application, in theory you pay only for what you use. Therein lies the catch though.

Paying for server calls

Finding out one piece of information (eg. the next time a bus will be at a certain stop) actually involves a whole series of server calls, not just one. With different numbers of server calls required for different tasks, this means that evaluating the cost of using the service is more complex than it seems at first and if your app requires a wide range of information requests on a regular basis, the price quickly racks up. Given the scale of the task at hand, it quickly became apparent that building the new busguru app solely using transportapi.com was not a viable option.

Patchwork solution

Instead, the ekreative team decided to use a patchwork approach. For London based requests (anticipated to remain the core base of users, at least when the nation wide update is first introduced) the information comes directly from TfL. For other UK cities transportapi’s amalgamating service is used, but with an important proviso: static data sets requested from transportapi get temporarily stored on a dedicated server. These sets contain information that changes on a much less regular basis than live bus times (eg. bus stop locations). By storing the information after a request is made, future requests for the same information are then answered from the dedicated server rather than by the expensive paid service, significantly cutting the apps running costs and enabling its viability. Of course even static data can potentially change and needs to be rechecked every now and then, but the savings enabled by this solution, partial though it may seem, are integral to the apps success.

Start using open data!

Creative uses of open data are always exciting to follow. It’s an area ripe with potential and in a state of constant development by those already taking advantage. Perhaps that’s you; are you constantly looking for new ways to squeeze more value out of publically available information? Are you hunting for app ideas that are genuinely useful? If so, comment below and leave us links to the most interesting public data apps you’ve found or developed so far. If it’s not you, now’s the time to get involved in the world of open data!