Sunday 9 December 2012

Web Apps and Pagination Queries

Paging controls are found on many popular web apps.
Amazon is a great example.
In a fair few web applications out there, you'll see the concept of pagination. That is you have a list of results and they'll be separated into pages which show a limited number of results (say 25 records per page). You can then usually go forward a page, go back a page or the first or last page. This is a common UI practice and I'm sure you've seen it before.

But, how is this implemented and what is the best way of doing so?

Well, the results shown on the page are usually held against a database. So the question quickly becomes:

"How can I query the database to bring back a given page of results?"

Now before I carry on lets first state that this blog post consists of database queries that have been run against an Oracle database (both the 10 and 11g versions). There's a strong possibility that they will not run against SQL Server, or any other database for that matter, but I suspect the general concepts remain the same. I should also state that I am by no means an Oracle expert. Yes, I know SQL reasonably well, but an expert? Far from it. So, if you find a better way of implementing the following features then it'd be great to here from you.

Now, back to the task in hand. We need to write a SQL command to run on an Oracle database that will bring back a "page" of data of a particular table and as an extra condition, the data returned must be in a given order. Let's also say that a page consists of 25 records.

You'll be surprised at how many ways there are to achieve this. In the 4.5 years that Qube have been using this functionality, our pagination queries have changed no less than four times. Some times we've made the odd small change to optimize the query, in other times we've re-written the whole thing and used a different technique. In each case, we've improved performance over the last iteration of changes. This is done with the goal of making our web application as fast as possible. Finally, we've now got to the stage where using one technique is faster on one table but slower than a different technique on another. This means there is no technique that's best for all tables. It'll depend entirely on the amount of data you have in each table, the indexes you've set up and the way in which your database is optimized.

I'm now going to take you through our little journey of finding the best possible pagination query. In all of the example queries I've highlighted the base query (the query which we want to paginate) in blue. Generally, the following methods wrap that base query up in order to implement paging. You'll need to change the base query to be the actual query you want to use pagination on.

Revision One - Retrieve All

So, our first option was to retrieve all records and let the application handle the pagination. So, if we have a table that consists of 100,000 records, we retrieve all records and then the application will display 25 at a time. The bonus of this method is it's extremely easy to implement. Also, moving from one page to the next is extremely quick, all the records are held in memory so navigating between them is simple and fast. There are however, some major downsides. For starters, retrieving all records is slow so the initial load time for the user is severely affected. If that's not bad enough (and it should be), the memory usage of such a technique is outrageously large, especially when you consider the fact that it's highly unlikely a user will want to navigate through the entire 100,000 records anyway, so you're using valuable memory on records that are never going to be shown. Not cool! Just with these two things in mind, this technique is going to scale very badly.

Example Query
SELECT COLUMN_ONE, COLUMN_TWO, COLUMN_THREE 
FROM TABLE_NAME 
ORDER BY PRIMARY_KEY

Revision Two - Nested Queries

Bringing back everything clearly isn't the way forward. So, with a little bit of use of ROWNUM (an Oracle feature which brings back the number of the row in a dataset, e.g. the first row has a number 1, the second has a number 2, etc.) we can bring back 25 rows at a time which will give us the paging functionality that we need.

Using this method the application doesn't need to store all the possible rows which lowers the memory consumption on the application significantly. It also improves the performance of the first page load. Instead of bringing back thousands of rows, the database only returns 25 rows and there's obvious performance implications for that. The downside however is that the performance of the next/last/first/last page buttons will be affected as the query will need to be re-run but for the next page of data. If all the rows were stored on the application from the original query then this additional database query wouldn't need to be run.

Example Query
SELECT * FROM (SELECT rownum as f2n_rownum, f2n_table.* 
               FROM (SELECT COLUMN_ONE,
                            COLUMN_TWO,
                            COLUMN_THREE 
                     FROM TABLE_NAME 
                     ORDER BY PRIMARY_KEY) f2n_table 
               WHERE rownum <= 25) 
WHERE f2n_rownum >= 1

Revision Three - Nested Queries Using WITH

As always, speed is key!
We've now got an implementation of our pagination query but is it the best implementation? Did you know that your average user expects your web application to load in two seconds or less and up to 40% of your users will leave your site if it hasn't responded after three seconds.1 That means these queries need to be as quick as possible, every second counts. When you're dealing with possibly thousands of records, it can be difficult to bring back results in that time frame.

So, we looked to see if we could improve the performance of our pagination query. If we can then that's a performance improvement across near all of our pages. As it turns out... there is a better way! Kind of.

We could improve the performance in two different ways. Firstly, we can use the WITH clause. This is known as subquery factoring.2 The second improvement is that we can tell the oracle optimizer how many rows we intend on using, this allows the optimizer to use this information to choose a faster explain plan for what we want. We do this by using optimizer hints in the query and in this hint we tell the optimizer that we want the first 25 rows brought back first (or however many rows are contained in your "page" of data). For more information on this try here: www.orafaq.com.

Example Query
SELECT /*+ FIRST_ROWS(25) */
       pageouter.* 
FROM (WITH page_query AS (SELECT COLUMN_ONE,
                                 COLUMN_TWO,
                                 COLUMN_THREE
                          FROM TABLE_NAME
                          ORDER BY PRIMARY_KEY
      SELECT page_query.*, 
             ROWNUM AS innerrownum 
      FROM page_query 
      WHERE rownum <= 25) pageouter 
WHERE pageouter.innerrownum >= 1

Revision Four - Nested Queries With ROW_NUMBER

Ok, so we're now using optimizer hints and we're using subquery factoring. All good stuff. But can we do more?

Well, we can. Kind of.

There is a SQL function called ROW_NUMBER(). It serves very much the same purpose as ROWNUM in Oracle but it works in both Oracle and SQL Server and oddly, when used can perform better than our previous queries but only in certain scenarios.

The problem here is I can't tell you why it performs better in certain scenarios, I can't even tell you in which scenarios it performs better but here is what I have found:

  • It seems to perform better than our previous methods if the query is modified to have a complex 'where' clause.
  • It seems to perform better than our previous methods if the data is ordered by a row that is not uniquely indexed.
  • The performance gains can be dramatic. In the previous examples, changing from one method to another may have seen an improvement ranging from nothing to a second or two. I've seen this method improve some queries by up to 5-8 seconds, especially on queries that order data by columns that aren't indexed.
Now I suspect all of this is very much dependent on the indexes you have set up on your tables, the amount of data in your tables, how you've got your database optimized and probably a  fair few other factors that I have no idea about so, the best way to know how this will perform for your queries is to test it.

Example Query
SELECT /*+ FIRST_ROWS(25) */
       *
FROM ( SELECT ROW_ONE,
              ROW_TWO,
              ROW_THREE,
              row_number() OVER(ORDER BY PRIMARY_KEY) innerrownum
       FROM   TABLE_NAME
     )
WHERE innerrownum BETWEEN 1 AND 25

Conclusion

I've shown you three different ways of implementing pagination within the database query. There are other ways which I haven't discussed. For example, you could follow this process:
  1. Run the query for ALL records (no paging) but insert the results of that query into a temporary table.
  2. Query that temporary table for the "page" of data that you want, using one of the methods above.
  3. When implementing the next/previous page function, you can then query the temporary table directly. 
Assuming that your original query isn't bringing back the entire table, you'll be selecting from a subset of the original data which should make the next/previous functionality faster. However, your original page load time will be slower as you'll need to insert the records into the temp table so, there's a trade off. 

I would imagine there's loads of other ways of doing this, if you find any that perform better than the above then let me know, it'd be great to hear from you!

And finally.... SQL Server

I couldn't end without mentioning the latest version of SQL Server and the good work Microsoft have been doing in this area. Microsoft have cottoned on to the fact that this paging functionality is now widely used and, as you can tell by this article, it isn't straight forward. They've gone out of their way to simplify this and built this functionality straight into the language making it very simple and, I would hope, a whole lot quicker than anything we can write in standard SQL.

I can't say I've had the pleasure of testing this but, according to the documentation, the feature is implemented by the introduction of two new keywords, OFFSET and FETCH NEXT and they're used in the following way:

SELECT COLUMN_ONE,
       COLUMN_TWO,
       COLUMN_THREE
FROM   TABLE_NAME
ORDER BY PRIMARY_KEY
OFFSET 0 ROWS
FETCH NEXT 25 ROWS ONLY

This tells the database to bring back the first 25 rows. To bring back the next page, you'd increase the offset by your page size (in our example, 25). For more info, check out raresql.com.

And it's that simple.

The sooner Oracle implement this functionality the better!


1 - Forrester Consulting, “eCommerce Web Site Performance Today: An Updated Look At Consumer Reaction To A Poor Online Shopping Experience” A commissioned study conducted on behalf of Akamai Technologies, Inc., August 17, 2009
2 - For more information on subquery factoring, see www.dba-oracle.com 

Saturday 29 September 2012

Browser Compatibility and Window Management

With more and more users turning to Chrome, Safari and Firefox for their web browsing experience and with the explosion of mobile devices now connecting to the web, the importance of making your web application accessible across browsers has never been greater.

With this in mind, I started to look into exactly what's required to make a web application accessible on all web browsers. There's plenty of blog posts on just this issue but most of them deal with differences in layout engines. In this post, I want to talk about window management namely because different browsers handle this issue completely differently and it's caught me out.

window.focus()

If your application launches new windows, maybe to show a data entry screen then at some point, you're probably going to use window.focus() in order to bring the new window to the front.

In Internet Explorer, this isn't the problem. Take the following code as an example:


<html>
<head>
<title>Untitled</title>
<script language="JavaScript" type="text/javascript">
<!--
   function windowFocus(){
      var wnd = window.open('','','width=200,height=100')
 
      window.onfocus = function(){ wnd.focus(); };
   }
//-->
</script>
</head>
<body>

<a href="javascript: windowFocus();">Launch Window</a>

</body>

</html>


The above code is pretty simple. When you click on the link, a new window should open. Should you try and navigate back to the original window then the new window should re-focus and be put back on top.This can be a handy function, if you absolutely need your user to respond to an event in the popup window, then you can force them too (or the user closes the window, which is responding in a way...). 

The problem is this doesn't work in Chrome. And it's not a bug. It's by design. I should add it does work in Safari so it's not a WebKit issue. Essentially, Google argue that you, the developer, should not be able to play around with windows on the desktop. That's something that should only be performed by a user. Personally, I don't agree. To see the actual bug report, check here and here

If anyone does find a way of getting this to work on Chrome, then it'd be great to hear from you. I unfortunately wasn't able to find a way.

window.open() and resizable

As it goes, the application that I'm working on does open new windows. These windows act as data entry screens and all of these windows are set to be a specific width and height. The reason for this is that we use absolute positioning on all the form controls in order to display them on screen. If we let the browser set the size of the window then it's more than likely that it'll set the size to something that'll either be too small, so scroll bars will be needed to see all of the controls, or, the window will be too large and there will be massive amounts of blank space shown. So, we set the width and height specifically to avoid these issues and then we disable the users ability to re-size the window.

To do this, when we call window.open, we use specific features that set up the window as describe, specifically we set the width, height and set the resizeable attribute to false. (See W3Schools for the window.open definition).

Here is a good example:


window.open('http://www.google.com','','resizable=0,width=200,height=100');


Again, in Internet Explorer, this all behaves exactly as you'd expect. However, in Chrome, Safari and Firefox, we have a different story. And again, it's not a bug, it's by design.

In Chrome, Safari and Firefox it's impossible to make a window non-resizeable. There's no way round this, it's just not possible. The argument is that by making a window non-resizeable, it's possible that the developer has some how messed up their screen layout, making it impossible for the user to see things that aren't on the screen. If you make the screen resizeable then you avoid this possible complication and if everything does render correctly then the user just doesn't have to resize the window and then it's no skin off off of anyone's nose. So, these browsers just ignore the resizable attribute.

I can see where the browser vendors are coming from, their argument does make sense, but, I also think it looks unprofessional if the content of a screen doesn't resize accordingly with a window resize. This then means this functionality needs to be built in to the application to maintain a professional looking user interface.

For more on this issue, check out Firefox's bug report on it found here: https://bugzilla.mozilla.org/show_bug.cgi?id=177838

Window Height

As I said before, this web application opens up windows of a specific width and height. This is achieved by using the width and height attributes of the window.open e.g.


window.open('http://www.google.com','','width=1000,height=700');


In Internet Explorer, the window opens at the specified width and height. In Chrome and Firefox, the same is true but in Safari, it's a different story. For some unknown reason, when the window opens, it opens with a height of 780px. Odd. No other browser behaves like this.

If you try to use window.resizeTo to fix the size, e.g.


window.resizeTo(1000, 700);


Once this is run, the window changes to a height of 620px. So it still doesn't behave as it should. This only seems to occur on Safari 5.1 (I haven't tested on any earlier versions). After a bit of research, I found that this is a bug, check here for more info: https://discussions.apple.com/thread/3341535?start=0&tstart=0

I find it absolutely amazing that a browser that is so recognized and so widely used suffers from such a simple bug. Again, annoyingly, there's nothing we can really do to solve this. In theory, you could loop over different parameters to window.resizeTo until the window is set to the height you require but it's far from an ideal solution. Hopefully, Apple will address this issue shortly and this problem will go away. But I wouldn't like to put money on it.

Anyway, those are a few of the experiences I've had with window management for web applications across differing web browsers. If I come across any more then I'll be sure to post them.

Enjoy.

Wednesday 1 August 2012

ExtJS - A Review


Around nine months ago, I was looking into a potential project for work. In it's simplest form, this project involved building a web application that would mostly be built using JavaScript and that worked on all the major browsers and in particular, it must be viewable on an iPad. So, I set about searching for a JavaScript framework that would support this.

Now I class myself as quite the geek and I like to think that I know of most of the major JavaScript frameworks. JQuery, Dojo, Prototype, etc. On top of that there's a few control libraries such as ComponentArt and Obout which I'm also quite familiar with. I looked into all of these but their functionality and look and feel didn't really jump out at me. I thought I'd do a bit of research and came across a framework that I'd never come across before... Sencha's ExtJS Framework. After spending the past couple of months working nearly exclusively with the framework, I thought it'd be a good idea to share my experience with you people so that if you're in a similar position to me, trying to decide whether or not to use an unfamiliar framework, you have a bit more extra information to go on.

It's worth mentioning that I'm in no way affiliated with Sencha so this is a completely independent review.

So, what do I look for in a good framework? Well, I think there's six categories to look at:
  1. Functionality - Unsurprisingly, a good framework needs to do what you need it to do. If it doesn't fit your requirements then there's no point in using it.
  2. Performance - There's not a whole lot of use in having a framework that does everything you need it to if it takes days to do it. How well does it perform?
  3. Ease of use - I would say I'm proficient in JavaScript. I'm certainly no expert. I expect a framework/library to be understandable and usable by someone with my level of understanding of the language. I do not want to have to take a two month extensive course in the inner workings of JavaScript in order to use the thing.
  4. Documentation - Any and every good framework/library needs to have good documentation. You don't want to be stepping through code to work out how to render something. It's much more effective to look up the documentation, have a quick read and then away you go.
  5. Extensibility - The chances are that at some point during your development cycle, you're going to need to add to the framework in order to achieve your desired functionality. How easy is this to do?
  6. Support - Sometimes you just can't work something out, is the support out there to help you through these difficult times?
So, how does the ExtJS framework compare in these categories?

Functionality

ExtJS has the most extensive functionality of any JavaScript framework I've ever come across. It has the ability to create some great user interfaces with some great special effects. Want pictures flying across the screen? No problem. Want something to fade in or out? Easy. And that's just for starters. It has a whole charting module: bar, line, graph, area, scatter, radar charts can all be created with minimal fuss. Along side that you've got grids which automatically support sorting, paging, editing and grouping.

Charts and graphs is only a small
piece of ExtJS functionality
ExtJS doesn't just do charting controls. It's has loads more to offer. Combo boxes which are a more enhanced version of the basic select HTML element are a selling point. There's also menu and toolbar controls which allow you to build user interfaces that will be familiar to your user. There are many more controls, just check out their examples here: http://www.sencha.com/products/extjs/examples/.

On top of all the controls, there's the functionality that the controls themselves bring to the table. Bar charts can just be rendered, or, you can animate them so they bounce in to position. Want to show a legend? No problem, that's in there too. Just about every time I've thought "I wonder if I can do x, y and z", I have a look and low and behold the framework supports it. It really is a very impressive framework.

Performance

With the amount of functionality that ExtJS does provide, performance is always going to be a talking point, especially when so much relies on manipulating the DOM which we all know can be slow. Now I've used ExtJS 4.0 extensively and I must say that the performance is very good. Like any programming library, there's way in which you can optimize your code so that you get the most out of it and I believe that if you do optimize then you'll be very impressed with its performance.

Performance of ExtJS4.0 is impressive!
There is a downside here though... ExtJS 4.1 was recently released and so I tried upgrading my project to use the new and "improved" library. After the upgrade, not only did certain controls just not work, the performance of my site seriously deteriorated. The whole reason for upgrading was to see if the new version improved performance, according to the promo by Sencha, it would do and although the 4.0 version wasn't exactly slow, if I could gain a bit more performance just by upgrading then it'd be stupid not too right? Well, I upgraded, spent a day fixing things that the new library broke and then spent another day optimizing the code and it was still slower than the 4.0 version. We're not talking about a few tenths of a second, we're talking  10's of seconds. At this point, I gave up and reverted back to the 4.0 version. Maybe it's possible to optimize it so it does run faster but with no real guarantee of that, I didn't feel I could justify spending the time in attempting it.

Ease of Use

ExtJS is exactly what I'd expect a good JavaScript framework to be in terms of ease of use. I simply picked it up and started to play around with it and soon got to grips with its functionality and how it can be used. Sencha provide loads of good examples ranging from the basic to the more advanced. All of their examples have their source code attached, you can simply look at the source code and work out what's going on. All of the objects and functions are logically named so the code is easy to follow. Put that together with a modern browsers JavaScript debugging capabilities and you can work out exactly how everything works and from there you can work out how to adjust the code to suit your needs.

If that isn't good enough, Sencha also provide well written tutorials on how to use the framework and how to deploy it, making it as simple as possible. If you want to check those out then have a look here.

Documentation
Documentation is a must for any framework
This is one area that ExtJS really does shine. The documentation for ExtJS really is excellent. Every method, property and configuration is well documented, just take a look here if you don't believe me. Not only do they provide documentation for the current version, they also have documentation for previous versions as well so you won't be looking at documentation that no longer applies!

You'd think this is one area that all frameworks would do well in, after all, documenting is just a case of writing down what everything does. You'd expect framework providers, especially ones that charge for use, would devote time to this area. However, you'll be surprised how poor the documentation is out there. jQuery is a framework that I've had a fair amount of experience with and they're pretty good with documentation, in contrast, I've also had a lot of experience with ComponentArt and they're documentation is absolutely shocking. I've wasted hours of my life trying to figure out how to use ComponentArt controls or trying to find out exactly what a certain property/method does because the documentation just simply doesn't tell you. Nine times out of ten, it's either non-existent or incomplete. You have no idea how much of a difference comprehensive documentation makes when you're trying to use a new framework.

Extensibility
Almost every time I've had to use a framework, it never quite works as I need it to so I end up modifying the source code or ideally, I build on top of the framework to add a bit of functionality to it. ExtJS recognises this and provide ways for you to override and extend objects. The framework heavily relies upon inheritance so extending and overriding controls is a very familiar concept for programmers that use object oriented concepts on a day to day basis.

Support
No matter how easy a framework is to use and no matter how good their documentation is I can guarantee at some point or the other you will come across a problem that you're struggling to solve. At this point you'll want to ask the wider world for a bit of help. Sencha have specific forums for this and for a fee you can access specific forums that are monitored by Sencha developers. ExtJS is fast becoming a widely used framework so, if you post a problem on a well known forum (say StackOverflow for example) then you'll be almost guaranteed a reply.

So... In Conclusion....

I think it's pretty fair to say that I'm a massive fan of the framework. This has blown frameworks that I have previously used out of the water. Some of the things I've managed to build in a day would have taken me months without the use of the framework. The documentation and general ease of use make developing applications with the framework a simple and effective task and because of the sheer quality of documentation, you feel your productivity increases just by using it. The thing I like the most... and this sounds kind of odd but, it just works. The quality of coding is very high. I've been using the framework on and off now for about nine months and I can count the number of problems I've come across on one hand and even then, I've been able to solve them with minimal effort.

It's not all good though, as I've stated previously, there is one big downside... upgrading and backwards compatibility. I upgraded from 4.0 to 4.1 and everything, and I do mean everything, broke. When I eventually got things working again the performance was so bad that I had to revert everything and stick with 4.0. I don't seem to be the only person that's had problems either. If you want to have a look for yourself, then you can check out the comments to Sencha's blog post about the 4.1 release here. Here's one comment that jumped out and me and summed up my frustration after a couple of days of trying to get things working after the upgrade:

"What a shocking release. An app that works without fault in 4.0.7 doesn't even show up properly in 4.1. We just got Sencha'd ... again!"

But that's the only fault I can find. If you're not worried about backwards compatibility and if the framework in its current state does what you need then I couldn't recommend it higher. It truly is a fantastic framework to work with. Just don't try upgrading.

Saturday 23 June 2012

Microsoft.Sharepoint.Sandbox.dll

Just a quick post about an error I recently came across, namely, this one:

"Could not load file or assembly 'Microsoft.Sharepoint.Sandbox, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c' or one of its dependencies. An attempt was made to load a program with an incorrect format."

So, how did I come across this?

I wanted to do some development on a SharePoint 2010 project but, I didn't want to do it on a server that had SharePoint installed (the code path used on a non-SharePoint server would mean that the SharePoint features wouldn't have been used).

To achieve this, I needed to copied across the Microsoft.Sharepoint.dll over to the non-SharePoint server and then have the project reference this DLL so that everything compiles correctly. I should add here that the server I copied the DLL from was a 64-bit server and the non-SharePoint server is a 32-bit server.

Upon compilation, everything is fine and dandy, everything compiles and there's no problems. However, at runtime, I get the above error. But why?

Well, there's two important parts to this error message...

Could not load file or assembly 'Microsoft.Sharepoint.Sandbox, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c' or one of its dependencies.

This tells us what file is causing the problem. Microsoft.SharePoint.Sandbox? We didn't include that! As it turns out, when you reference the Microsoft.Sharepoint.dll, upon building the project, this is automatically put in to your web applications bin directory.

The second part tells us why it's causing us a problem...

An attempt was made to load a program with an incorrect format.

This occurs when you try to run a 64-bit DLL on a 32-bit process. 

So, we now know what's causing the problem and why. How do we solve this problem? Simple. We delete it! On a non-Sharepoint server, the Microsoft.Sharepoint.Sandbox.dll isn't used so it can be removed from the bin directory of your web application. The best way to do this is to add a post-build event to your project which deletes the DLL so you don't have to do anything. Obviously however, if this is used on a SharePoint server you will need that DLL so just be aware of that.

And that's it, another odd problem solved.

Tuesday 12 June 2012

HTML5 - Geolocation

After my last blog post, I thought I'd take a look and see what this geolocation melarky is all about.

As it goes the API is extremely simple and easy to use which makes blogging about it pretty straightforward.

The aim of the API is to get the longitude and latitude of the device that is accessing your website. This information can then be fed into another application to make it more user friendly. For example, a common use for this information is to put it into Google Maps to show exactly where your user is.

So, how do we use this API?

First off we need to make sure that the browser supports geolocation. To do that, we simply need to check that the navigator.geolocation object exists. That can be achieved with the following:


if(navigator.geolocation){
   // Do Geolocation Stuff
}


All straightforward so far. So now it's time to grab the longitude and lattitude. There are two methods available within the API to accomplish this: getCurrentPosition and watchPosition. getCurrentPosition runs once and gives you the users location. watchPosition will continually run, allowing you to track the user as they move. Both run asynchronously. There is one final method included in the API, clearWatch. This method allows you to stop tracking the users position after you've called watchPosition.

Now let's have a look at these methods in a bit more detail.

getCurrentPosition

The function definition:

getCurrentPosition(successCallback, errorCallback, options)


The success and error callback parameters explain themselves. The options parameter allows the developer to specify the timeout value, the maximum age of a position call (basically, how long it lives in a cache) and how accurate the position call will be.

To determine how accurate the position call will be you need to understand how geolocation works. There are a variety of ways in which a users location can be worked out, as listed below.

  • WiFi - Accurate to 20 metres. The MAC address of any and all WiFi access points detected by the users device is sent across the web and mapped against a database of wifi access points and locations. From what I can find out, different browsers use different databases. For example, Chrome and Firefox use a database created by Google. Safari however uses Skyhook's Wireless service.
  • IP - Accuracy varies greatly and can give false information. In this instance, an IP address, or a range of IP addresses are mapped to locations within a database. Unfortunately, due to the nature of IP, addresses and locations change and occasionally IP addresses can get mapped to a completely wrong location.
  • GPS - Accurate to 10 metres. This method only works outside so is only ever an option with a mobile device. GPS is quite an advanced system, if you're interested in the gritty details of how it works, have a read of this. This method unfortunately can take a bit of time and can use more power which may be important if you're target device is a mobile phone/tablet. It is however, the most accurate out of all of the geolocation methods.
  • GSM/CDMA IDs - Accurate to 1km. This will take unique ID of your mobile device and will use it against the local mobile phone towers to triangulate your position. Accuracy will significantly improve where there are more towers, so, if you're in an urban area, accuracy will be far greater than if you're in a rural area.
Ok, so, here's an example of using the getCurrentPosition method:


var success = function(position){ 
    alert("Latitude: " + position.coords.latitude + ", Longitude: " + position.coords.longitude); 
};

var error = function(error){
    alert("Error. Cannot find your position");
};

navigator.geolocation.getCurrentPosition(success, error, { enableHighAccuracy: true });


The enableHighAccuracy option essentially means that the method of determining the users position that provides the highest accuracy will be used. This only really matters when your user is using a mobile device with GPS. GPS uses more battery power than the other methods and can take a couple of minutes to return so may be something you want to avoid where possible.

watchPosition
The function definition:

watchPosition(successCallback, errorCallback, options)


Look familiar? It should do. It's exactly the same as the getCurrentPosition method, with one small difference. This method will return a watch ID. The watch id is then passed into the clearWatch method to stop tracking the users position, but more on that in a bit.

So, like I said, the purpose of this method is to track the users position. When they move, assuming that no errors are thrown, your successCallback will be called. This callback takes the exact same parameters as the getCurrentPosition successCallback function did so, you can grab the new longitude and latitude co-ordinates and use them for whatever you need.

Just as a very quick example of the use of this:


var success = function(position){ 
    alert("Latitude: " + position.coords.latitude + ", Longitude: " + position.coords.longitude); 
};

var error = function(error){
    alert("Error. Cannot find your position");
};

watchId = navigator.geolocation.watchPosition(success, error, { enableHighAccuracy: true });


clearWatch
The function definition:

clearWatch(watchId)


Pretty simple eh? This effectively cancels your watchPosition call made earlier. So, as an example, you could have something like this:


var success = function(position){ 
    alert("Latitude: " + position.coords.latitude + ", Longitude: " + position.coords.longitude); 
};

var error = function(error){
    alert("Error. Cannot find your position");
};

watchId = navigator.geolocation.watchPosition(success, error, { enableHighAccuracy: true });


function stopTracking(){
  if(watchId != null){
    navigator.geolocation.stopWatch(watchId);
  }
}


You can then call the stopTracking function from wherever you want.

And that's about it for the API. If I'm honest, its simplicity surprised me. I personally think that location based services will really take off in the next year (if they haven't already) as HTML5 becomes the standard on more and more browsers. Location information opens up a world of possibilities, from helping you to find your friends in a crowd to providing more relevant information to you when you're performing web searches. Now that the technology is there, it's up to us developers to use it to give our users the best possible experience.

Before I sign off though, I should mention one thing regarding location services and it involves one of the 'keywords' being spoken about all the time at the moment.... privacy.


Google Chrome prompting the user.
Obviously, tracking a user's current position does come with a few privacy concerns. Web browsers have done what they can to address this by ensuring that when you request a users position, either via the getCurrentPosition or the watchPosition method, that the user is notified and they are then given the option of denying you access to that data. (If this occurs, your error callback will fire rather than your success callback.) However, this in my eyes isn't enough. I think websites should take a bit of responsibility too!

Ok, here comes a bit of a rant so feel free to stop reading here but a pet peeve of mine is sites that do request my location information and then use it for something that isn't stated on the tin. Take for example Facebook: Now and again I may want to publish where I am in the world so my friends can see for whatever reason. So, when I go to publish my location the website prompts me that Facebook wants to access my location data. Not a problem, how else are they going to publish my current location? What I do not then expect is for my location to be posted on everything afterwards. Especially when I've gone to the pub instead of heading home! Facebook doesn't just do it after you've published your location - it updates your location whenever you make a comment or a status update and as far as I'm aware, there's no easy way stopping that from occurring.

Now I'm sure Facebook has some setting somewhere that lets me turn it off for posts but I shouldn't have to. I agreed to let Facebook access my data to do a specific task that could not be accomplished without that data. I did not agree to let Facebook plaster it over every post/comment I subsequently make. I'm sure Facebook isn't the only culprit here and I understand that if you throw up a configuration option for each and every action then it's going to become pretty tedious. But, if I had been notified of their intentions at the start then I may not be ranting.

Ok, rant over. In conclusion to that little out-burst... just make sure your users know why you need their location data and exactly what you're doing with it. If not for your users benefit, then do it for your own as I'm sure someone will be ranting about your website if you don't.

Monday 4 June 2012

The Mobile Web - To App or Not To App, That Is The Question


Accessing the web on the move is becoming something all of us are doing more and more of each and every day. With smartphones and tablets becoming more advanced on an almost daily basis, the question of "should I make my website mobile enabled?" has been replaced with "how do I make my website mobile enabled?" and with technology in it's current state, that question can be boiled down to the following:

"App or Website?"

By this I mean, do you write a native app which you can download from an app store and run on your mobile device or, do you improve your website so that it can be accessed by a mobile device easily?
In my humble opinion, and bare in mind I am primarily a web application developer, that the website option is the way to go. Most things can be achieved by a website and it reaches a far wider audience than an app ever would do.

Why do I say this? Well, I'm going to go through what I consider to be the important factors when weighing up the above question, hopefully after that all will become clear.

Audience Reach

This is the first major issue and one that should hold some serious weight. If you develop an app, your app will only work on a specific platform. So, if you build an app for the iOS then only apple devices will be able to use it. What if your user is using an Andriod, Blackberry, Samsung or some other smartphone? Well tough. You've limited the amount of people your product/service can reach straight away. The workaround for this is to build the app separately for each individual platform. That in it self brings problems. If you're a software house, it's unlikely that you're going to have developers who know how to program for iOs, Android, Windows Phone 7, etc. That means you're going to have to train them up, that takes time and money. If you're not a software house, or you don't want to build it yourself, you can always outsource the work, but that's not cheap. Put simply, unless you're a company with a large amount of resources or, you're somebody with an immense amount of time on your hands, that's not really an option.

Let's compare that to making your website platform independant. Yes, over the years, making a website work on many different browsers has been a complete headache but things are changing. Browsers are learning! Yes, the odd tweak is needed here and there but in the most part, if you program to W3C standards then you shouldn't have many problems now days, especially on the major mobile platforms, which in my eyes, doesn't include Microsoft Internet Explorer. So you've made your website, you've programmed to W3C standards, what next? Well nothing. That's it. All browsers on all phones can access your mobile website. That's considerably cheaper and easier in just about every way you can think of.

Look and Feel

When discussing this topic with fellow developers, people inevitably say that a website won't give the same look and feel as a native app. "You can't get rid of the address bar!" I hear them say. Well, that's not strictly true. You can, but both you, the developer and your user need to know what to do. In iOS (I own an iPhone and have access to an iPad so know this platform reasonably well) you can add meta tags that, when run from the home screen, will get rid of the address bar and will let you style the status bar too. For example, the tag:


<meta name="apple-mobile-web-app-capable" content="yes" />


will ensure that the web application loads in full screen, with no address bar, when loaded from the home screen. Unfortunately though, there is a bit of a flaw in this. As I said, this only occurs when loaded from the home screen. That means your user must first add the website to their home screen. As far as I'm aware, there's no way of providing the user with an easy way of doing this, they must be familar with the iOS user interface. I'm sure this is equally true with Android phones. This does pose a bit of a problem, maybe it's not a problem for your particular product/service but it is something to bear in mind.

A Website Can't Be Viewed Offline!
"A website can't be viewed offline, what if my user is on a plane or in tunnel?". I hear that argument a lot when talking about this and it just doesn't hold up. HTML5 has introduced Offline Web Applications which I've blogged about before. They allow a website to be viewed with no internet connection available. Yes, it assumes you've visited the site previously but you can easily argue the same with an app, after all you need to be online in the first place to download the app. And yes, I agree, offline web applications can be a pain to create, especially if your content is dynamic but they're by no means impossible. I imagine it'd be no more harder than creating an app from scratch. Again though, it depends on your needs. If you definiately require offline access then it can be achieved by both an app and by an offline web application. Chances are though, you don't.

Access To Phone Devices
Unfortunately, building a website isn't the holy grail. After all, if it was there wouldn't be a need for apps. This is one area where apps have the upper hand. They have access to your phones devices, for example, your phone has a microphone. Your app can use that. Your phone has a camera, your app can make use of that. Now in theory, if you're just deciding whether to make your website an app or to make it mobile compatible, then this may not be a problem. Your original website wouldn't have had access to these devices either but maybe by using these controls, you can make a certain process within your website easier. It's something to consider.

I should say on this note that phone developers are beginning to realise this and seem to be building API's that help bridge the gap for websites. In the iOS 4.1 update, Apple sneaked in an undocumented update to it's Safari browser... two new JavaScript events were created, ondevicemotion and ondeviceorientation (see http://mobile.dzone.com/articles/how-use-gyroscope-your-iphone for more details). These provide website developers access to the accelerometer data and the gyroscope data of an iPhone/iPad which they can then use for whatever they want. 

The App Store

Just about all mobile devices now have some form of "App Store". A central place that a user can go to search and download your app. Whether the use of an app store is a good or bad thing completely depends on your situation. The app store does give users a central place to be able to search for your specific app. It may also open up another revenue stream for you; you can charge people to download your app! On the flip side of the coin, it's an extra expense, for example, you need to pay Apple to be able to put your app in the app store. On top of this, all apps are accredited by Apple so you need to make sure your app ticks all of their boxes. This may or may not be extra work for you and your development team. Plus, accrediation takes time. You've got to wait for someone at Apple to do their job before your app gets out in to the world. Maybe this extra time delay isn't a problem, maybe it is. Either way, It's certainly something that you don't have to worry about when building a web application.

The other thing to consider in this area is, how do you push out updates? If your application is sold to many different clients, it's quite possible that each client will buy and use a particular version of your product. What happens when you need to release an update? Maybe some clients have contracts so that they can download the latest updates, maybe some don't. There's no way to distinguish. In it's current form, the app store (at least the Apple version) doesn't support this kind of business model, either all users have the ability to download the latest update, or none of them do. This may very well lead to clients receving access to versions of your product that firstly they haven't paid for and secondly, and arguably more importantly, they may not want. As always, there are ways around this, you could for example release a "new" app for each client. You'd then have to come up with a way of managing that. If you then have that across all major mobile devices (I'm thinking Android, iOS, Blackberry, Windows), you've now got to have some form of version control for all of those, for all your clients. That's going to get very messy, very quickly. Maybe you could build your app so that it's backwards compatible and then only allow new content to be accessed by the clients that have that in their contract? There are ways around it but the chances are, it's not how you do things currently and will require a bit of a re-think in your release procedures.

So, what do you do?

In my very humble opinion, if your application is a web application, first and foremost, make sure it works on mobile devices before even thinking about building a native app. The amount of effort and resources that are required to make sure that your app has the same audience reach as a web application just isn't worth the bother. A native app, in my eyes, should be considered an extra bonus and nothing more.

As an example, let's look at the National Rail website as I think it's a particularily good example. Using their website, I can find out the times of trains across the whole of England. If I access http://www.nationalrail.co.uk on my desktop then I get the full blown website, I can look up times, book tickets, look up services updates, the lot. If I access the site on my iPhone or iPad (these are the only devices I have access to at the moment), then a specific mobile device website is loaded. The website I see there is a bit more stripped down but, I can perform all the same functions with minimal fuss. 

Once National Rail had that in place, they went one step further and built an app. If you load the app on the iPhone, like the mobile specific website, it too allows you to look up train times. The difference here is that they've taken advantage of device specific features. For example, the iPhone has the ability to track your position via GPS. National Rail have then used that information within their app to give you a listing of the nearest stations to your current location, something not really possible with current web technology (That's not strictly true, you could use the new HTML5 Geolocation API but I have no idea how accurate that actually is... maybe a subject for a different blog post).

Ok, so what if your application isn't a web application at all, maybe it's a desktop application that you're thinking of porting to a mobile device. Well, there is no generic solution for this. It'll depend on your product and the service you provide. As a general rule of thumb, I think it's always better to port your application to the web before creating an app but then again I am a web developer so my opinion may be a little biased.

I hope the above has helped but this is a subject that has quite a few different schools of thought so I'm interested in hearing what you think so feel free to comment!

Tuesday 15 May 2012

IE, JavaScript and the Story of the Weeping Angels


I came across a very odd problem the other day in the way in which Internet Explorer handles DOM items with an ID.

Take the following piece of HTML for an example.


<html>
<head><title></title></head>
<body>
<div id="testElement" style="width: 100px; height: 100px; border: 1px solid black;"></div>
</body>
</html>


You can't get much simplier than that. Now say you want to access testElement and change the width of the element. You'd probably do that using the following piece of JavaScript code:


document.getElementById('testElement').style.width = '200px';


All very straightforward so far. There is another way of doing this though, one which isn't recommended but is supported by all the major browsers. You can simply write:


testElement.style.width = '200px';


If an element in your HTML has an ID, the browser will automatically put it in the window scope so you can access it directly. No need for document.getElementById. Cool eh?

Well, it turns out Internet Explorer supports this little feature in a bit of an odd way. Take the following HTML page:


<html>
<head><title></title>
<script language="JavaScript" type="text/javascript">
<!--
  TestObject = function() {
  this.id = 'TestObject';
  }
//-->
</script>
</head>
<body>
<div id="testElement"></div>
<script language="JavaScript" type="text/javascript">
// alert(testElement.id); // We'll uncomment this line a bit later.
window.testElement = new TestObject();
alert(testElement.id);
alert(window.testElement.id);
</script>
</body>
</html>


What you've done here is create a DOM element with an id of testElement. So, the browser should have created a window.testElement variable that'll give you the appropriate DOM element when accessed. You've then explitically defined the testElement variable to be a new TestObject. So in theory, when the first and second alert is shown, the testElement variable should be pointing at our TestObject. The id should therefore be 'TestObject'. In both alert boxes, 'TestObject' should be displayed.

When you run the above, that's exactly what happens. No big surprise there.

Ok, now uncomment the commented line. What I'd expect here is that the first box should display "testElement" as that's the id of the DOM element. You then assign the TestObject to testElement so, when the second and third alert box is shown, you'd expect to see "TestObject".

When you run the above, the first alert box displays 'testElement'. Good so far. The second alert box displays 'testElement'. Eh? That's surely wrong. The third alert box displays 'TestObject'. What? How can window.testElement and testElement be pointing at different things? They're the same variable! Comment the line again and everything goes back to normal. How can this be?!


Weeping Angels!

For you Doctor Who fans, you'll know what I'm talking about when I talk about Weeping Angels, but for those who have no idea, a weeping angel is a creature that, when looked at, automatically turns to stone. When not being viewed, they go about their usual business. It's a good analogy for this behaviour because, after a bit of experimenting, I found that as soon as you look at the testElement variable, it's at that point that the browser actually points the variable at the DOM element and makes it read only. This means that if you reference the variable anywhere, then it'll affect what your code is actually doing. Even if you're debugging and place a watch on the variable, it'll have the same effect. These kind of variables, in my book, are about as ugly as a weeping angel, just see the above picture for an example.

I should say, only Internet Explorer (I tested on IE9) seems to handle DOM variables like this. The above code behaves exactly as you'd expect in both Chrome and Firefox.

So, how to avoid this? As most JavaScript programmers know, programming in the global (window) scope is just bad practice, for a variety of reasons but the main one is doing so can lead to naming conflicts pretty easily, especially if you're using third party libraries. This problem re-affirms this. It is a naming conflict, just not in the traditional sense as the browser is doing some of the work for you. Anyway, if you avoid programming in global scope then you won't come across this problem. Unfortunately, from time to time, it's unavoidable, especially if the problem is actually caused by a third party library, like in my case. In these cases, as you saw before, if you reference the variable using window.variableName, then it seems that that will always point to your object, not the DOM item, which should hopefully give the behaviour that you want.

Enjoy!