Image Cutter 1.42 and Google Maps API v3

Version 1.42 of the Image Cutter software was released yesterday. This includes the addition of a template to allow the use of Google’s API v3. This version of the Google Maps API doesn’t require an API key and works better than v2 does on mobile devices like iPhones, iPads and Android tablets.

You can download the latest version at the following link: Image Cutter 1.42

Trackernet: Where are all the Tube Trains?

This is starting to become obsessive, but I can’t help wondering how many trains are running on the London Underground and where they all are. The Trackernet web service released by TfL allows you to see all the running boards for stations on a line, but doesn’t tell you where all the trains are. I did an earlier post about just the Victoria line trains, but I’ve now built this into a web service that works out locations for trains on the whole network.


Trains on the London Underground network for 11:30am on 30th November 2011

The map colours follow the normal line colours, so District (Green), Victoria (Light Blue), Central (Red), Northern (Black), Bakerloo (Brown), Jubilee (Grey), Piccadilly (Dark Blue), Waterloo and City (light green). Note that Circle and Hammersmith and City are all shown as yellow and there are no pink markers on the map. This is because the Trackernet API does not distinguish between Circle and Hammersmith and City trains and both lines are queried in one web request, so they’re difficult to separate out.

The idea is to build this into a web service and publish it on MapTube as a real-time Tube map. Using the locations of trains and the time to station information we can build a model of whether a line is running normally and where delays are occurring.

The basic technique behind how the positions are calculated relies on using the time to station information from the running boards at every station on the route to find the minimum time for every unique train. This is then taken as the most accurate location estimate and its position interpolated between the last and next stations based on the time. It is actually a lot harder to work out which line a train is on due to the fact that multiple lines can share platforms at the same station. For example, query the Piccadilly line and the District line and the resulting data will contain Barons Court for both, so you have to separate out the Piccadilly trains and the District trains and make sure you don’t count the same ones twice.

Now that the code can handle the Underground network, the next steps are to do the same for National Rail, London Buses and London River Services.

If you’re interested in live train data, it’s also worth looking at the following site that was created by Matthew Somerville:

FBX Exporters Part 4

In the previous three parts, I outlined the plan for getting geometry from MapTube via C Sharp into an FBX file using the C++ SDK provided by Autodesk. This final part shows data in a world map exported from MapTube and imported into 3DS Max.

The above image shows the world countries 2010 outline from MapTube. There a few countries missing as there was no data for them in the original dataset, so they show as blank on a MapTube map and their geometry is not exported. This is more visible in the perspective view where you can see the holes in Africa and the Middle East:

The exported geometry will eventually be coloured in the same way as MapTube, but for the moment all the geometry objects have a green material assigned to them.

The final export file can be downloaded from the following link: MyFBXExport


It’s worth pointing out that I’ve had a number of problems with the Quicktime FBX plugin that comes with the Autodesk FBX SDK. It seems to crash every time I close it and when displaying the above file there are some significant problems with how it renders the geometry. Most notably around the Hudson Bay area in Canada, parts of Europe and much of Russia. As it displays fine in Max, I can only assume this is a limitation of the Quicktime FBX renderer. I’ve also had to do some re-scaling of the geometry as it is exported in the Google Mercator projection, using metres. This means that the numbers are too big for Max to handle, so I’ve had to rescale them.

To recap on how this process works, here are the development steps needed to achieve it:

1. A C Sharp program which is a modification of the MapTubeD tile rendering procedure reads the geometry and data from the MapTube server and returns it as an iteration of SqlGeometry objects.

2. Each SqlGeometry object is simplified using the Reduce operation as we don’t need the full level of detail in the output FBX file.

3. The C Sharp program uses native methods in a C++ DLL, which I’ve written to control the operation of the FBX exporter in Autodesk’s SDK. A handle to the FBX document and scene that we want to create is obtained and then a native “AddGeometry” function is used on every geometry object until a final “WriteFile” function is called. The geometry is passed using the OGC well known binary format which is an efficient way of passing large blocks of complex geometry and is also independent of byte ordering.

The DLL which does the actual export is a 32 bit program with functions exported using C names rather than decorated C++ names to make it easy to link to the C Sharp function stubs. Internally, I’m using the GEOS library to parse the well known binary geometry, extract polygons and write the points to the FBX scene hierarchy.

That’s the proof of concept to demonstrate that this method works. The aim now is to see what we can do with geographic data now that we have the ability to load it into art tools like Max and Maya, game engines like Unity, or frameworks like XNA.



FBX Exporters Part 3

The first two parts of the FBX export process dealt with getting the FBX SDK working and exporting some simple geometry. Now what’s required is the ability to pass complex geometry from the MapTube side using C# over to the FBX side using C++. The obvious way to do this is to pass the geometry in the OGC well known binary format (WKB), so I’ve been looking at GEOS which is a C++ port of the Java Topology Suite (JTS). I’ve managed to use this in conjunction with the FBX exporter to create simple geometry from WKT which I’ve loaded into 3DS Max.

One of the problems I had was building a debug version of GEOS version 3.3.1 as the instructions aren’t quite right. The make command for a debug build is:


As I’m using Visual Studio 2008, I had to run “autogen.bat” first to create the required header files, and also make sure I do a clean between the release build and the debug build. Once this library was built successfully, I could use the WKT reader to read in some test geometry and build an FBX exporter around it.

[c language=”++”]
string version = geos::geom::geosversion();
//geom::Geometry* geos::io::WKBReader::read ( std::istream & is );
std::string poly("POLYGON ((30 10 0, 10 20 10, 20 40 20, 40 40 30, 30 10 0))");
WKTReader* reader = new WKTReader();
Geometry* geom = reader->read(poly);
delete reader;

The entire FBX exporter is too big to replicate here, but the part that extracts the geometry from the GEOS geometry object and creates the FBX control points is as follows:

[c language=”++”]
//create control points
int NumPoints = geom->getNumPoints();
KFbxVector4* lControlPoints = lMesh->GetControlPoints();
CoordinateSequence* coords = geom->getCoordinates();
for (int i=0; i<NumPoints; i++)
lControlPoints[i]=KFbxVector4(coords->getOrdinate(i,0), coords->getOrdinate(i,1), coords->getOrdinate(i,2) );

The only other thing I’ve done is to create a material for the polygon so it shows up as red in 3DS Max.

Now I’ve demonstrated that all the component parts work, the final stage of getting geometry from MapTube into 3DS Max will be to write a C++ library on top of the FBX exporter and GEOS which can be used as a native library from C#.

FBX Exporters Part 2

In the first FBX exporter post I got to the point where the export of simple geometry from one of the Autodesk SDK examples could be loaded by the Quicktime plugin. This used the SDK as a multithreaded statically linked library which I used with one of the examples to create a plane object. The following image shows a more complicated file containing a marker (red), custom geometry in the form of a cube (grey) and a camera (looks like a camera).

The code to get to this point is rather complicated, but I copied the UI Examples CubeCreator example program supplied with the SDK which showed how to set up the cube mesh with all the correct normals and textures.

The scene graph is set up with a camera, marker and mesh as follows:

[c language=”++”]
// build a minimum scene graph
KFbxNode* lRootNode = pScene->GetRootNode();
// Add the mesh node to the root node in the scene.

The creation of the mesh object prior to this is a lot more complicated:

[c language=”++”]
// Define the eight corners of the cube.
// The cube spans from
// -5 to 5 along the X axis
// 0 to 10 along the Y axis
// -5 to 5 along the Z axis
KFbxVector4 lControlPoint0(-5, 0, 5);
KFbxVector4 lControlPoint1(5, 0, 5);
KFbxVector4 lControlPoint2(5, 10, 5);
KFbxVector4 lControlPoint3(-5, 10, 5);
KFbxVector4 lControlPoint4(-5, 0, -5);
KFbxVector4 lControlPoint5(5, 0, -5);
KFbxVector4 lControlPoint6(5, 10, -5);
KFbxVector4 lControlPoint7(-5, 10, -5);

KFbxVector4 lNormalXPos(1, 0, 0);
KFbxVector4 lNormalXNeg(-1, 0, 0);
KFbxVector4 lNormalYPos(0, 1, 0);
KFbxVector4 lNormalYNeg(0, -1, 0);
KFbxVector4 lNormalZPos(0, 0, 1);
KFbxVector4 lNormalZNeg(0, 0, -1);

// Initialize the control point array of the mesh.
KFbxVector4* lControlPoints = lMesh->GetControlPoints();
// Define each face of the cube.
// Face 1
lControlPoints[0] = lControlPoint0;
lControlPoints[1] = lControlPoint1;
lControlPoints[2] = lControlPoint2;
lControlPoints[3] = lControlPoint3;
// Face 2
lControlPoints[4] = lControlPoint1;
lControlPoints[5] = lControlPoint5;
lControlPoints[6] = lControlPoint6;
lControlPoints[7] = lControlPoint2;
// Face 3
lControlPoints[8] = lControlPoint5;
lControlPoints[9] = lControlPoint4;
lControlPoints[10] = lControlPoint7;
lControlPoints[11] = lControlPoint6;
// Face 4
lControlPoints[12] = lControlPoint4;
lControlPoints[13] = lControlPoint0;
lControlPoints[14] = lControlPoint3;
lControlPoints[15] = lControlPoint7;
// Face 5
lControlPoints[16] = lControlPoint3;
lControlPoints[17] = lControlPoint2;
lControlPoints[18] = lControlPoint6;
lControlPoints[19] = lControlPoint7;
// Face 6
lControlPoints[20] = lControlPoint1;
lControlPoints[21] = lControlPoint0;
lControlPoints[22] = lControlPoint4;
lControlPoints[23] = lControlPoint5;

// We want to have one normal for each vertex (or control point),
// so we set the mapping mode to eBY_CONTROL_POINT.
KFbxGeometryElementNormal* lGeometryElementNormal= lMesh->CreateElementNormal();


// Set the normal values for every control point.


// Array of polygon vertices.
int lPolygonVertices[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,12, 13,
14, 15, 16, 17, 18, 19, 20, 21, 22, 23 };

// Create UV for Diffuse channel.
KFbxGeometryElementUV* lUVDiffuseElement = lMesh->CreateElementUV( "DiffuseUV");
K_ASSERT( lUVDiffuseElement != NULL);

KFbxVector2 lVectors0(0, 0);
KFbxVector2 lVectors1(1, 0);
KFbxVector2 lVectors2(1, 1);
KFbxVector2 lVectors3(0, 1);


//Now we have set the UVs as eINDEX_TO_DIRECT reference and in eBY_POLYGON_VERTEX mapping mode
//we must update the size of the index array.

// Create polygons. Assign texture and texture UV indices.
for(int i = 0; i < 6; i++)
// all faces of the cube have the same texture
lMesh->BeginPolygon(-1, -1, -1, false);

for(int j = 0; j < 4; j++)
// Control point index
lMesh->AddPolygon(lPolygonVertices[i*4 + j]);

// update the index array of the UVs that map the texture to the face
lUVDiffuseElement->GetIndexArray().SetAt(i*4+j, j);



So, we have to define the vertices (control points in the language of the SDK), normals and UV coordinates for the mesh to show in the Quicktime viewer. It’s also worth mentioning that I’ve had to force the output FBX file from the exporter to be in binary format as the viewer refuses to load the ASCII format FBX. In addition to this, I’m still getting application crashes when I close the Quicktime viewer.

Now I have the ability to create custom geometry, the next step is to write an interface to allow me to pass geographic data to the exporter via C#. After giving this some thought, the obvious solution is to pass well-known binary (WKB) from the C# program to the C++ library as a block of bytes. This is a relatively easy format to produce and decode into geometry, so shouldn’t take long to write.

Part three will deal with the mechanics of getting actual geometry to the exporter and generating an FBX file from real geographic data.

Exporting Geographic Data in FBX Files

I’ve been looking at how to export the geographic information contained in a MapTube map into an art tool like 3DS Max or Maya. The reason for this is firstly to make it easier to produce high quality geographic presentations, but also, by employing a recognised art tool chain, we can also get the data into 3D visualisation systems built around XNA (XBox) or Unity.

Originally, I was going to implement a 3DS exporter as this is a well-used format that would allow geometry to be imported by Google Sketchup, Blender, or a long list of professional art tools. After coming across Autodesk’s FBX SDK, I decided to create an FBX exporter instead. Although this is a format that can’t be loaded by either Sketchup or Blender, the SDK is quite flexible and can also export Collada (DAE) and Wavefront OBJ files which the free tools can import. In addition to this, it can be imported by both Unity and XNA.

Autodesk supply a viewer plugin for Quicktime, but I had some problems getting this to work with my first export attempts. The example below shows a simple screenshot:

Although a flat black plane on a grey background isn’t fantastic for a first attempt, it took a while to get this far as the examples don’t tell you that the Quicktime viewer doesn’t like ASCII format FBX files and you have to change the example format to BINARY.

[c language=”++”]
//altered export from ASCII to Binary
int lFormatIndex, lFormatCount = pSdkManager->GetIOPluginRegistry()->GetWriterFormatCount();

for (lFormatIndex=0; lFormatIndex<lFormatCount; lFormatIndex++)
if (pSdkManager->GetIOPluginRegistry()->WriterIsFBX(lFormatIndex))
KString lDesc =pSdkManager->GetIOPluginRegistry()->GetWriterFormatDescription(lFormatIndex);
printf("%s\n",lDesc.Buffer()); //print out format strings
//char *lASCII = "ascii";
char *lBinary = "binary";
if (lDesc.Find(/*lASCII*/lBinary)>=0)
pFileFormat = lFormatIndex;

This is a copy of the “ExportDocument” example that comes with the SDK, but with the type changed to binary to allow it to load.

The next problem is learning how to create my own geometry and figuring out a way of connecting the native C++ library to the managed C# code used by MapTube. My initial thought was to create a managed wrapper for the FBX SDK and use marshalling, but, on further examination of the SDK, it’s much too complicated to do in any reasonable amount of time. So, plan B is to write the code to do the export as a native C++ process, expose enough methods to allow this to be controlled through marshalling and interop via the C# code and do the FBX export through that route. This only depends on being able to marshall the large amount of geometry data, but this should be possible to work out.

After these first experiments, it’s looking like the pattern will be something like a reader/writer object with a choice of export formats as FBX, Collada or OBJ to allow the assets to be loaded into as many art packages as possible.

The next post will cover the generation of the geometry and its export to FBX.

Trackernet: The Victoria Line

I’ve been meaning to look at TfL’s Trackernet API for a while now. It works through a REST based web service which gives access to all the London Underground running boards on a line by line basis. You issue an http request of the form:

and the result is an XML file containing train information for every station on the Victoria Line. Substitute “B” instead of “V” and you get the Bakerloo line instead. I had managed to figure out a way to get approximate train locations when the Victoria Line got suspended one morning, so I couldn’t resist looking to see where all the trains had ended up:

According to my data, there are 25 trains on the line. The way the positions are calculated is quite complicated as the original information comes from the running boards for every station and the time to platform estimates. Trains are uniquely identified through a train number and a set number as a composite key. I simply iterate through all the data for every station and take the lowest time to station for every train, which gives me the train’s next station. Then I use the location code provided by the API and the time to station estimate to interpolate between the last station and the next station.

One feature worth noting is that because the time to station is given for every station along the train’s whole route, you can use the data to build up a dataset of the time required to travel between any pair of stations. Also, because the information is processed from the running boards, the program should be able to process National Rail train locations from the information on their website.

By using only the information provided in the XML response from the API means that I am able to construct a web service that doesn’t require any state information to be retained between calls. In addition to this, it doesn’t require any knowledge of the tube network and how the stations are connected together.

This is still very much a prototype, but once it’s working for all the lines, it will be released as a real-time feed on MapTube.

Twitter Maps and #ukriots

With all the media hype surrounding the use of social networking and the London riots, it left me wondering what was actually being said on Twitter in the UK. It was also a good opportunity to test out the new MapTube map creation software which can handle 35,000 clickable points on a map with ease (we tested with 500,000). So, the aim was to create a map of UK tweets which I could explore around the areas where the riots were happening to see what people were saying about them on Twitter.

In order to do this, I used the Twitter client that Steven Gray wrote to collect geocoded tweets from Twitter. This has been used for things like the real-time heatmap of London 2012 #1yeartogo tweets:

The resulting map can be seen below:

Tweets captured from Twitter between 15:00 and 22:00 on Tuesday 9th August 2011. See text for further details. (link to live map)

Once the data had been collected as a CSV file containing “UserId”, “Time”, “Tweet”, “lat” and “lon”, the processing was done using Excel. This will feature as a separate blog post in more detail, but I created columns of riot related hashtags and cleanup related hashtags. Then these were combined into a colour code based on whether the tweet is a general tweet (Blue), contained a riot tag (red), contained a cleanup tag (green) or both riot and cleanup tags (yellow).

There are full details in the “more information” link on the live map, but I collected 34,314 geocoded tweets in the period, of which 1,330 contained riot hashtags and 87 contained cleanup hashtags.

What’s interesting about this map is that I was expecting more tweeting about the riots. 13,330 is less than 4% of all geocoded tweets.

Where this map really comes into its own is the ability to click on messages around the riot areas and see what people are saying. I think what this highlights is that you need some sort of natural language processing as there is obviously a lot of discussion about the riots not using hash tags. People are tweeting that they or their children are scared as it sounds really close, or tweeting to people to tell them they have arrived home safely.

The other interesting thing for natural language processing researchers is that Twitter has a language all of its own. When you start reading some of the tweets it’s obvious that the contractions and slang that is being used will be a challenge to understand.

One thing I need to fix in MapTube is that it returns too much information on the popup when you click on a location. The point and click functionality returns all points covering the area of your click. If you are zoomed out a long way, then this can be hundreds of points on the speech bubble popup which causes the client browser a number of problems. I think limiting to around 20 returned points would be a safer option.

Making Maps From Wikipedia

I’ve been looking at web-based sources of geographic data and Wikipedia links are something I’ve wanted to try out for a while. I found the following page containing Worldwide fossil sites:

List of fossil sites

This gives a list of sites, but with no locations:

The “Site” column contains href links which can be followed to pages like the following:

The coordinates can just be made out in the top right hand corner of the page.

As all Wikipedia pages follow a common theme, the coodinates are embedded in a <span> tag with class=”geo”. I already had a Java program for loading a web page and converting it into xhtml, so I used this to turn the original list page into a csv file by extracting the data out of the html tables. One of the columns in this file contained the links to the site-specific pages, so another program was written to follow all these links and extract the location from the site page.

While the general technique works, only about 25% of the data had links to pages with lat/lon coordinates, so the final map is somewhat incomplete, but this can be edited manually. The map itself was built using MapTube’s new map creation system which works using the CSV file of data. The final map can be viewed at the following link:

Fossil Sites

New MapTube Map Creation Feature

We released a new feature on the MapTube website today which will make it easier to create new maps from data in CSV files. The underlying technology is used on the SurveyMapper site and for other real-time visualisations like which shows tweets using the #1yeartogo hashtag for the London 2012 Olympics.

Creating a map of abandoned vehicles from the London Datastore using MapTube

The new update to MapTube adds a graphical user interface which allows the user to upload a data file, choose a colour scale and publish the map on MapTube directly. One of the driving forces behind this was the idea that creating a map should be simple enough that you could do it using an iPad. Data on the London Datastore  is in the correct format, so you can copy the CSV link directly from the site, which is exactly what has been done in the above image. I’ve created a YouTube clip showing the whole process, which can be viewed at the following link:

This feature is still experimental, but at the m0ment it handles point data in lat/lon coordinates (WGS84) or OS coordinates for the UK (OSGB36). Point data can be drawn using markers, or as a heatmap showing point density. For area data, one column in the data is selected as a key field and this is joined with the geographic data stored in MapTube’s database to draw the map. For example, using the following data:

We have four columns: Constituency, Party, PartyCode and Change. In the CSV file the first line must be the column headings, then every subsequent line contains data. The CSV file would contain the following:

Aberavon,LAB,1,LAB Hold
Aberconwy,CON,2,CON Gain

The “Constituency” column is the area key in this case, but MapTube determines this automatically when the CSV file is loaded, along with the type of geography, which is Parliamentary Constituencies. In order to colour the map, numeric data is required, so in this example, a column labelled “PartyCode” has been added where “LAB”=1, “”CON”=2, LD=”3” etc.

The colour scale is then chosen and the finished map submitted to MapTube where it can be viewed along with any of the other maps. There are help pages accessible through the ‘i’ icon on each section which contain further information.

As mentioned before, this feature is still experimental and we will be gradually adding more geographic data to the MapTube database to allow maps to be built from additional geographies. The aim is for MapTube to be able to automatically detect the geography just by analysing the data and, at the moment, the following geographies can be used:

Government Office Regions (UK) (GOR)
Lower level super output areas (UK) (LSOA)
Medium level super output areas (UK) (MSOA)
Output Areas (UK) (OA)
Postcode Districts (UK) (PostcodeDistricts)
County and Unitary Authority (UK) (CountyUA and ONSCountyUA)
Districts (UK) (Districts and ONSDistricts)
Census Area Wards (UK) (CASWards)
World Borders 2010 (WorldBorders2010ISO2 and ISO3 using the ISO country codes)
Parliamentary Constituencies 2010 (UK) (PCON2010)

US States and Zip code areas will be added shortly, along with adminsitrative and Census boundaries for other parts of the World.