Creating real time web GIS applications - part2

Creating real time web GIS applications - part2

Date: Jan. 21, 2014, 12:35 p.m. 
Author: Ivan Pasic
Category: GIS 

In my last article I was speaking about creating real-time web gis application for tracking some GPS devices.
The main idea was to use websockets as a communication between client and server. Every time when server receives POST request data from GPS device it sends message to client (browser) and the latest position is updated on map.
However, I didn't speak much about some problems that you might encounter and there are some things that I believe are worth mentioning.

First, the problem that we might have is how to handle all that received GPS logs. 
For example, if you've had GPS device which would send you it's location each 10seconds (6 per minute) then after 8 hours of tracking you would have 6*60*8=2880 rows in your table.
Well maybe it's not so many right now. But imagine that you have your rent-a-car company with e.g. 20 cars and you want to track them all. In that case you would have 57 600 entries in database for just 8 hours of each car's driving. As you can imagine, after some time that could cause some problems (maybe not with saving them but definitely with querying them).
As I'm still exploring this real-time tracking stuff I'm not quite sure what's the best way to solve this problem but here are 2 ideas that come to my mind.
1) Don't save your data in database
Very simple. If you don't need to look history of your device movement then you don't need to save your data to database at all. All you need is to make your server send latest received position to client with which he has open websocket connection and that's all.
You can make your server send that data to client in whatever format you want (JSON seems like good solution). 
You would still have real-time tracking app and see where is your GPS device at some time. Sometimes that's all we need.

2) Save only some GPS logs to your database
First, I must say that I'm not 100% sure that this is right approach but it seems to me like reasonable idea.
So, the idea is to save just some of the received data (lat/lon) and then according to them interpolate the line between those coordinates which would represent some device's movement. For example, you could save position and time for 1 of 10 received logs and then create lines from last to latest saved log. E.g. instead of saving points P1, P2, P3, P4, P5, P6, P7, P8, P9, P10 you could save just P1 and P10 and then create Line between P1 and P10. Then you could save that line to database and have 1 row in table instead of 10. 
With that approach, you would still have some simplified path of your vehicle that is saved in database and that you can query and look at it later.
However, as I've already said, I didn't test this approach yet so don't take it for granted. :)

Let's move on.

The second problem that we might encounter with receiveing data by POST request from GPS device is number of requests per second that our system can handle.
Again, if you track small number of devices at the same time then it wouldn't be problem. But imagine that you have some tracking system where you need to track 1000 devices at the same time. If you'd have 1000 GPS devices that send you position each 10seconds (6/min or 0.1/sec) then you would averagely have 100 post requests per second (req/sec) which is not such a small number (unless you have more servers for your app).
We need to be sure that our system can handle that number of requests. That's why we should perform load testing while we build our system.
Load testing is method for testing our system to see how it behaves when certain number of users are using it at some moment. It's actually  way to measure our system performance based on a number of users.

There are many ways and tools you can use to perform load testing but I've recently discovered locust.ioand I'm really more then satisfied with it. It's based on pyhton and it's very simple to use.
As it says on it's official website: "Locust supports running load tests distributed over multiple machines, and can therefore be used to simulate millions of simultaneous users".
Great thing about Locust is that you can simulate user behaviour and beside that swarming process is monitored from a web UI in real-time.
If you want to learn more about it then check their official documentationhttp://docs.locust.io/en/latest/index.html

Here is simple load testing that I performed with locust for the app I've described to you in my last article.
*Note: Although it's too early to test our app beacuse it's far away from finished normal app I still wanted to try it and to see how locust works. :)

locustfile.py

from locust import Locust, TaskSet, task

class WebsiteTask(TaskSet):
        @task(1)
        def get_index(self):
                self.client.get("/")

       @task(10)
       def send_position(self):
               self.client.post("/location", {"lat":"23.45", "lon":"16.87"})

class User(Locust):
        task_set = WebsiteTask
        min_wait = 1000
        max_wait = 1000

And here are my results for 100 simulated users and hatch rate 1

As you can see, for 38.8 post requests per second and 4.1 get request per second my system is working pretty well and there are no failures or errors (there are more post requests then get requests because in my code I've set higher weight argument for task that do post request).

However, if I'd increase number of simulated users or hatch rate there would probably be some errors.

But as I've said, this is just prototype of some serious app so I did this load test more to present you locust then to really perfrom some testing of our system.