The Blog

Faster Data Input using Form Navigation

Project SwiftCount Checklist Entry Form

Data entry is at the heart of the operations at Project SwiftCount. Our data clerks have to fill forms like the one in the screenshot above over and over again. In studying how they accomplish this task, we noticed a certain activity that takes a lot of time – leaving the keyboard to use the mouse.

Every time a data clerk wants to navigate from one input field to the next, they have to reach out to the mouse and click on the next field. The really smart ones had figured out the tab key and use that to move from one field to the next. You’ll agree with me that if you have to fill in fifty of those input fields, that’s a lot of work.

In order to help, we worked on a jQuery snippet that allows the use of keys on the keyboard to navigate from one field to the next. So all the data clerk needs to do is either press the “n” key on the keyboard to navigate to the next input field or “p” to navigate to the previous. This works for us since numerals are the only valid inputs in each of these text boxes so trapping the “n” and “p” keys to perform the navigation suffices.

We’ve created this snippet demonstrating how it was done.

Two are better than one when it comes to servers

Two (and probably more) are better than one when it comes to deploying servers for a web application. When we deploy our applications, we generally like to separate the database server from the application server. If we were deploying a PHP web application for instance, the web server (which also doubles as the application server in the simplest cases) will be deployed separately from the MySQL database server. That is going to cost more in terms of hardware resources but we’ve found out that it not only improves the application performance but it also increases the application reliability. At TimbaObjects, we obsess over the visual appeal, speed, data integrity and reliability of our applications and so our approach to building such applications generally means we separating data processing from data storage.

In early hours of this morning, one of our application servers which is being used to store, manage and process data for one of our clients went down. Our client is an accredited elections observation group and they recruit and train their election observers to fill out and send answers to an elections checklist (using structured text messages) to a shortcode which is then routed to an application we built that parses, validates and stores the data in this structured text message. It then sends out a response as a reply to the sender.

Our connectivity with the telecom operators is via an SMPP link with our content aggregators and that’s a good thing because if our server goes down for any reason, messages bound for our shortcode get aggregated until we re-establish the connection. This is about the second time we’ve had this problem in the close-to-one-year period this application has been live. Further investigations into the crash revealed that we ran out of memory on the box and while the operating system was attempting to salvage more memory by killing tasks, it seg-faulted. Unfortunately, our watchdog is unable to detect this condition and reboot the server so a reboot had to be done manually. This was an occurrence only happening on the application server; the database server has been largely unscathed – we have an uptime of over 303 days on the database server (as at the writing of this post).

Our deployment uses an Apache server with mod_wsgi to serve the application and preforks 10 processes to serve requests. In order to provide access to previous elections data, we copy over this configuration for every election. With a current deployment covering five elections schedules, we have over 50 apache processes. With a server deployed with only 768MB of RAM, that’s a lot of memory we’re talking about here. The machine on which our web server is deployed also doubles as the SMS processing server. So we have an additional Kannel bearerbox and smsbox processes routing messages over HTTP to a node.js HTTP proxy server we built to handle HTTP requests on behalf of RapidSMS (which our application is built on). Add all this processes together and you have a box that’s about to go belly up.

We solved the problem by reducing the number of processes apache forks to handle requests for the archived applications since those applications don’t get used that much. From about 59 apache processes, it went down to 25. We’re currently using 56% of available RAM now to serve the applications and no swap yet. With a swap size of only 256MB, if we had something much bigger, we probably wouldn’t have run out of memory.

Of course we could have just used one server with a lot of RAM to serve both database and application servers, however, a lot of things can go wrong and the fewer applications you have running on a server, the better for its stability. You simply don’t want to have a single point of failure in your application architecture.

Next time, we’re considering separating the SMS processing application from the application server so in the event that the application server goes down, text messages will continue being processed.

Changing the future

You’ve probably seen a movie or two about someone having the power to predict the future or in some cases see into the future. It’s all too likely that if you were able to see into the future, you’ll change a thing or two – and then the future changes.

In the Jan. 2, 2012 edition of The New Yorker, there’s an IBM ad with the title: “Drivers now ‘see’ traffic jams before they happen” citing a case in which IBM helps Singaporean motorists see traffic congestions one hour before they happen.

This and many types of other prediction applications use tons of historic and realtime data to accomplish this feat. The utility of this application is undoubtedly undisputed (pardon the pun). If the removal of fuel subsidy doesn’t discourage people from driving their own cars and hence potentially reduce congestion in a city like Lagos, such an application will be very useful.

Someone might ask, if prediction algorithms are able to tell us there’ll be heavy traffic on 3rd Mainland bridge, then the suggestion might be to take the Ikorodu road route. However, that then causes a problem on Ikorodu road since every motorist will then want to take that route. Good question, but then the algorithm also uses realtime data and would put into consideration the fact that traffic is building up on Ikorodu road and hence make another prediction and hence a new suggestion. It’s like having a traffic warden who sees all the routes at once and can make better decisions as a result of this oversight.

Welcome to the new year.

Freedom’s new helpmeet

When the Time Person of the Year 2011 award went to ‘the Protester’ I was not surprised. Despite the presence of other nominees like Steve Jobs (who needs no introduction) and Elizabeth Warren (a US Senate candidate), I still was not surprised.

It began in Tunisia, where a 26-year-old street vendor named Mohamed Bouazizi drenched himself in paint thinner and lit a match after harassment and seizure of his fruit scales by council officials. It is said to be the final straw that broke the provabial camel’s back and ignited a series of protests and revolutions that spread through out the Arab world and unto other countries across the globe. But I digress.

Instead of plugging in the headphones, entering an Internet-induced fugue state and quietly giving in to hopelessness, protesters used the Internet to find one another and take to the streets to insist on fairness and (in the Arab world) freedom. For instance in Tunisia,  a blogger recorded the protests with his BlackBerry, uploaded the video to Youtube, and it got half a million views in a day. Hours later, President Ben Ali flew to exile in Saudi Arabia. After just four weeks, they protesters had won.

The Egyptians had their own Mohamed Bouazizi: an underemployed middle-class 28-year-old named Khaled Said. Sometime in 2010, after apparently hacking a police officer’s cell phone and lifting a video of officers displaying drugs and stacks of cash, he was arrested and beaten to death. Wael Ghonim, then a 29-year-old Google executive, created a Facebook page called We Are All Khaled Said to memorialize him. It went viral! In January, Ghonim returned from Dubai to Egypt met online and then, face to face with other protesters and the rest is history.

In Madrid, Spain  Olmo Gálvez, 31, an Internet entrepreneur just back from three years working in China and new to politics helped set up social-media networks for the protest.

The Gaddafi led government saw that the Internet is a valuable wartime resource, like a critical bridge over which supplies can flow. As long as you can deny it to your enemy, you don’t blow it up — you keep it intact for your own use. And so they went for it just like Egypt did. After Gaddafi shut down all communications coming from the Libyan General Telecommunications Authority, rebels developed a new strategy that includes a new network that enabled them to communicate and access the Internet. They launched Free Libyana on the 2nd of April 2011. And by April 12th 2011, a span of ten days, Opposition said it had an amazing 725,000 subscribers. Again. the rest is history.

So, I was far from amazed when rumors began to spread, during the Occupy Nigeria protests, that BIS was going to be suspended by Telcos in the country. My timeline literarily caught fire! “Fools! have they never heard of Twitter?” “As long as there is internet, we will communicate” “We’ll use sms and phone calls” “Let the Telcos also pack up and leave!” just to mention a few of the reactions I saw. Someone actually ‘approached’ me on Twitter to ask if TimbaObjects could hack together something that would be sms based to ensure the Occupy Nigeria protests would be seen and heard by the whole world and keep communication amongst protesters alive. Guys had gotten smart!

The Internet is super important and has proven even more so being the main tool behind the social, political and economic revolution across the globe. So no matter if broadband access is a human right, protected as a mechanism for free speech or completely corrupted to serve the status quo, it’s also wise to have a backup plan.

When some future governments are backed into a corner by a popular uprising supported by Internet communication, they will probably reach the same conclusions that Libya and Egypt did: reestablish control over national communications at any cost, and pick up the pieces later. That’s why the Internet is too vital to be left in the hands of centralized authority, and it’s why you should press for more diverse Internet connectivity wherever you happen to live.

Freedom has wooed and won the Internet’s heart and she has brought along her smart kids: Social Networks. They do love their new daddy.

Long Live Freedom and it’s new Helpmeet!

NSE Data API usage report

It’s been two weeks since we launched our Nigerian Stock Exchange data API. We decided to do some analytics on the API usage yesterday and to our amazement, we’ve received a total of 665 hits on our API since launch on July 13th, 2011. Certainly some of this requests don’t translate into valid requests. Here’s a breakdown:

Response Type Response Code Hits
OK (Successful API requests) 200 410
OK (All successful requests) 200 439
Forbidden (Unauthenticated Requests) 403 9
Not Found 404 182

Breakdown of HTTP Request Types

The large number of Not Found requests were generated by trojans making requests like /PhpMyAdmin and other such requests (they were probably looking for security loopholes).

Of the 23 API tokens that have been provisioned, we have a 65% usage on those keys. Here’s a distribution of the requests we’ve received from those keys:

Requests Users %
1 4 26%
2 4 26%
3 1 7%
4 2 13%
5 2 13%
8 1 7%
369 1 7%

Requests Vs API Users

In order to make the points in the chart fit within a reasonable chart size, the topmost dot was reduced.

How to pitch to VCs

Delegates at the Pitch Monday Event

Yesterday at Pitch Monday, a number of startup founders in Nigeria had an opportunity to pitch their ideas at investors and get a chance to raise funding for their ideas.

While most of the ideas that were pitched were notable, it was quite obvious that most of the founders weren’t prepared for their pitches or didn’t know how to.

There are numerous articles online on how to prepare for your pitch and so I’ll recommend reading as much on this subject as possible.

In this blog post, I’ll be sharing and expanding some of the tips Tomi Davies shared with the delegates on pitching their ideas at the tail end of the event.

Tomi Davies summarized by saying pitching an idea is all about telling a story and your story needs to contain certain key components for it to be effective.

He breaks it down into the following key components: Proposition, Organization, Economics and Technicalities (POET).

Proposition

Your proposition has to do with you describing what your service is all about; what problem you are attempting to solve; what value you intend to create and for whom. Depending on how much time you have, you also want to talk about your suppliers, customers and how your product (or service) is different from the competition and your strategy for maintaining your competitive edge.

Organization

In organization, you essentially describe your team. Who the members on your team are (or will be) and what roles they play (or will play). It does appear to me that investors favor startup teams of two or more founders – preferably a team that comprises of someone who’s good at product development and marketing (the business guy), the hacker (developer) and the aesthetics (interface) team members.

Economics

So this is where a number of startup founders fumble. It’s important to show the investor the money. Most investors are not investing for philanthropy but for profit and if they cannot clearly see how your startup will make them money, you just wouldn’t get them interested enough to invest in your startup. The economics of your startup outlines your cost centers and expenditure (both operational and capital) and your revenue sources. You will do good to explain how you intend to realistically make money in your startup and show how long an investor is likely to wait to recoup their investment and move on to profit from your startup. This is where you talk about things like the return on investment and your rate of return.

Technicalities

This has to do with how your startup is meant to work. Most startup founders start with this but as Tomi Davies mentioned yesterday, this should be the last thing you talk about. If you’re discussing your startup with other geeks, sure this can come first but not when you’re pitching to investors.

In conclusion, I’m really excited about the startup community in Nigeria and I’m looking forward to seeing startup founders grow their businesses into world-class companies.

Your Titanic, My Titanic

Am sure we’ve all seen the movie ‘Titanic’. And in the event you’ve been living in a crevice somewhere in the Himalayas, it a re-enactment of the fateful voyage of the world’s largest sea vessel of its time (Ask someone else for the romance in the story. I’ll suggest Joy). Well, the ship sank 4 days into its maiden voyage and went to the bottom of the Atlantic taking about 1,517 souls along. History. The 1997 movie was a blast and stayed on top if the box office grossing charts for 12 years only recently taken down by Avatar.

Lets talk numbers. If you hate maths, never mind, I’ll bring it down to you. It cost White Star Line $7.5 million to put the Titanic to sea in 1912. Yep, quite a sum back then. Guess how much the movie ‘Titanic’ grossed? About $1.84 billion. Yep! Billion with a  B. Let’s take that in for a moment. (*leans back and smiles with eyes closed*)

Ah! Do i see a hand at the back of the class? Inflation you ask? Well, even after adjusting for inflation and bringing both figures to a chosen year, it’ll have cost $28.86 million to build the liner and the movie would have grossed $383.34 million! Yep! Now, class, a simple division yields 13.28 (to 2 s.f.). QED.

My point? The story of the failure of the ocean liner, Titanic, made about 13 times what it cost the owners to actually build it. White Star Line suffers a huge loss, James Cameron picks up the tale, underlines the romance with a bunch of good actors and with the 21st Century and other studios made a success out of a shipwreck.

The moral of the story?

Technology is meant to make life, in general, easier, worth the while and longer (in some cases). Many argue nothing new can be invented that only advancements can be made. It is only logical to try and solve the same old issues (even pre-1912) with cutting edge technology. Especially with Information and Communication Technology.

Technology drives commerce. According to the story line, no one saw the huge ocean liner sink in 1912, but with 1997’s technology, millions lined up to the Titanic sinking afresh! It’s time we looked at those age old ‘shipwrecks’ from a different angle, sprinkle in some innovation and find a way with today’s technology. That’s why we are here.

Dance for Data

I was to take a bike from the highway to the “Mast”. I’d been told that my brother-in-law-to-be’s apartment was in the block of flats directly across from a telecommunication mast. I was passing through Ibadan and it was getting rather late, so I decided to spend the night in my sister’s fiancee’s place and get back on the road at the crack of dawn.

After a 6 minute ride, I was there. Standing before the mast, I wondered how many such monuments to our developmental progress in the sphere of telecommunications were scattered over the federation. They had to be thousands and thousands of them. In my mind’s eye, I saw a technical executive’s office at NCC with the map of Nigeria stabbed in a thousand places indicating where masts stood.

It was however not the mast itself that caught my attention, it was the muffled sound from the sound-proof generator installed to power the site. There was a fence and a security guard whom I later learnt had bought a motorcycle from the proceeds he made billing the mobile phone users in the neighborhood for each hour their phones spent plugged into a power source at his security post. He always had light, they seldom did. Demand and supply. Economics 101.

The Telcos got their licenses in 2001 at a cost of about N31.9billon each. Add to that the cost of doing business, (e.g. having to pay the security dude at their mast who’s ‘selling’ their energy at a premium and keeping it for himself. Even with people like him on the payroll, there have been countless cases of theft and vandalism at such mast sites nationwide). Ok, yes, the costs are huge. But they dare not claim annual loses! Profits are still staggering, simply put. With EBITDA figures that sound like astronomical distances.

But it’s still a wonder that Indonesians pay about $7 for BIS monthly and we pay about $20. Data is still very expensive here and most folks I know with Maemos, Androids and iOSs dare not let their mobiles loose in the byte store. It’s a quick way to commit financial suicide. A few get data plans and these are still very expensive. A friend ran an N8000 data plan monthly on his Galaxy S. Sad much, I tell ya!

 

 

To whom shall we go for data on our smartphones? We have to dumb them down to make them affordable. switch off most Apps, increase the time lags between retrievals and other such barbaric stunts. I recall when I got my first smartphone (Bless it’s soul *wipes a tear*), anytime folks were checking it out, I had to stand guard. Not because of privacy concerns but because, one innocent click on an App or widget and zzzaaaaaaap! my balance would read something thing like N0.09 in few seconds. Happened a few times.

The Red Indians had a dance ritual they performed when they needed the rains. I wonder if there’s a Nigerian dance ritual that would help bring us a torrent of data. My hope is that it’s as easy as alanta.

Private beta announcement of NSE data service

For well over a year now, we’ve been testing an API we developed at TimbaObjects for providing data on trades at the Nigerian Stock Exchange. A number of our clients already use this data in production for some of their applications and we’re repurposing this API for public use.

This data service is the first of a number of such data services that we intend to provide to developers who will in turn use this to create valuable applications. With this NSE data API, you get access to data on all stocks that were traded on the Nigerian Stock Exchange on a daily basis.

We’re yet to launch this on a private beta basis and we’re seeking feedback in the form of a survey on what kind of services or applications will be built using this API. This will give us a sense of the data requirements and help us provide more value to developers who are likely to use this service.

To gain access to this API on launch, please fill our survey and don’t forget to leave your contact information and we’ll sign you up when we launch.

[Survey link]

Score Brokers and the Laws of Historical Precedence

“Where were you in 2008 when we flogged this same team 1-0 in the semi-finals?” Kunle yelled. I was on his side in this argument, but he was going into the past as he was fond of and this irked me on end. The guys on the other side of the divide in this soccer argument were going to start going into ‘the books’ too to pull out records, at that stage it would be over for me. Kunle would have to go it alone. Am sorry.

I am a soccer fan, a jersey donning-display picture-die hard fan of  Manchester United FC, reigning champs of the EPL (no apologies to all other EPL team fans, you all can stop reading now). I will defend my team at any soccer argument ‘tourney’, but please in the name of all you hold dear, do not ask me for historical precedences. I do not recall all the history neither do I believe those will count in the present. These ‘Score Brokers’ actually recalled all the stuff and believed it mattered, even if it happened 30 years ago! They’ll tell you the likely outcome in figures! Puleaseeeeee! Are you shitting me? Count me out.

Kunle is one hell of a Score Broker. His day job? A stock analyst and broker. He ran a website to service his clients and sent them SMS reports along side. But his passion was soccer. His team? My team! *winks* he sides with the best. But before I get carried away, back to why I’m writing this. We sat and watched the match together, well, our team lost (don’t ask me for details! Hatters!), but as we took the long walk back home from the viewing center, I got thinking about all the historical precedence that had been discussed and how it really hadn’t favored us and alas! the match too had not. Did this thing really work?

Fans like Kunle could go as far back as the 1960s to site cases like lawyers! I suddenly asked Kunle “When was the NSE formed?” He looked at me strange, still wallowing in the defeat of our club side. I guess he wasn’t expecting such a question, a UEFA related question would have been welcomed. He did answer, “It was established in 1960, trading actually began in 1961. It was called the Lagos Stock Exchange at the time.” I left it at that.

A few days ago, I had the joy to call Kunle and ask him how he would feel if we could deliver to to him all the stock quotes of the NSE not only of the present and fairly recent, but also past quotes. Well, his reaction is best imagined. I almost dropped my phone. With such a database and the tools of his trade, trends would stand out like whores in a convent. Trends was what all the historical precedence was all about. Trends were the guide to forecasting and navigating the shifting weather and waters of Stock brokerage and analysis.

This is exciting and I like the vibes we are getting from the experts in that field. Our team at TO? We are ready to play. Do I hear someone say mobile? Gosh! You folks never have enough. Heheheheheh!

From my desk this Thursday evening.