Megalith

For some time now I have been working on no less than five major projects:

Database querying application
Database CRUD front end
Statistical analysis application
Data modelling application
Business Intelligence application

I was aware that there was a certain amount of overlap and was thinking of ways to reduce repetitive code, but today I had an epiphany. That's not some fancy ice cream, just the realisation that all these projects could be, and indeed should be, merged into one.

So Megalith is born. Whatever you can do with data, you'll be able to do it with Megalith.

Watch this space!

Dashboards? Reports? What's wrong with an export to Excel?

One of the Big Things at the moment is the Dashboard - a screenfull of controls (graphs, instruments or whatever, the more colourful and eye-catching the better) which provide a snapshot of the company's data at a given point in time. Now these can be very useful if they are sensibly and thoughtfully designed, but often the design is more about aesthetics than practicality, resulting in a solution which is at best useless and at worst dangerous. It is easy to get into the habit of looking at the dashboard regularly but briefly and if the pretty pictures look good then the company and life in general is good. But unless the dashboard is exceptionally well thought out they could easily be hiding no end of problems.

So the solution? Like it says in the title, an export to Excel. And I think most people actually recognise this; the one thing I have heard from users more than anything else is "can we have that in a spreadsheet please?". What people want is to be in control of their own data. Over the past ten to twenty years most people who work in offices have built up a very considerable amount of expertise in using Excel, and given a chunk of raw data can manipulate it to get exactly what they need, often very quickly and intuitively.

The JavaScript Story

Over the past 6 months I have been doing something that even a year ago I wouldn't have dreamed off - coding in JavaScript!

When I started I very naively assumed it was a sort of "Java for Web Pages", complete with the full complement of built-in support for encapsulation, inheritance and polymorphism, so it has been a bit of a shock to discover how limited the core JavaScript capabilities are, as well as being pretty weird from the point of view of somebody like me coming from a C/C++/C# background.

Despite that I have made some good progress using JS for its traditional purpose of manipulating the DOM, and also in AJAX. The next step, of course, is to use JS for more ambitious applications with the type of functionality found in desktop applications. This is where the limitations of JavaScript start to kick in and make life difficult. There are numerous workarounds to emulate interfaces, inheritance etc., but they are just that - workarounds. Of course you can write a library of use a ready-made one to provide such functionality but, however you do it, you are still using very fragile and inefficient extra code to simulate the capabilities provided as a core part of other languages. A lot of people believe HTML5/JavaScript applications are the future but, with the current state of JavaScript and the fragmented and rather motley collection of JS libraries and frameworks I find it hard to believe how that can happen for a very long time.

People have been saying for perhaps a decade that eventually desktop and web applications will merge, and HTML5 and the recent advances in JavaScript engines and libraries represent a huge advance in that direction, but unless there is something truly radical on the horizon then I can't see how this can happen for another decade or so.

So where does this leave my web development efforts? Well I will certainly be continuing to work on HTML5 and JavaScript to push the envelope of what web apps can do, but as for any ideas I might have had a few months ago of desktop-like plugin-free apps running in a browser - err, no, not quite yet!

No place for amateur software engineers

There has been a lot floating around the meme pool recently about how more people - possibly even everyone - should learn to code. The most high-profile manifestation of this is the Mayor of New York taking an online JavaScript course.

At one time, maybe fifteen years ago, you were either a programmer or you weren't. Then people started playing with VBA in Excel. They created entire database systems in Access. They got a free copy of Visual Basic CCE on a disc taped to the front of a magazine. The free Express versions of C# and VB .NET came along. Now half the population are programmers.

I'd like to be able to say this is a good thing but, honestly, it isn't. However much modern development environments abstract away the complexity of software development, that underlying complexity is still there and is actually very much greater than ever.

To tap away or drag-and-drop away without understanding or even being aware of that complexity is just the path to chaos. There are many fundamental concepts that "proper" programmers don't even consciously think about, they are just hardwired into their brains, but they are crucial to their everyday work. But the amateur programmer, blissfully unaware of this, cannot help but create inelegant, inefficient, unfathomable and unmaintainable code, whilst being intensely proud of what is probably the most complex thing they have done in their lives.

You might argue, quite reasonably, that the sort of development done by non-professionals if of the type that would never get done at all by professional developers, and that a far from perfect system is better than none at all. This may well be, but the scope of many systems grows rapidly, and a small system put together by somebody in accounts or marketing or wherever just for their own use gains interest from colleagues who ask to use it or for it to be expanded. Somebody else asks for it to be integrated with some other system, perhaps the company's "official" system. The system's proud parent runs out of talent and then the IT department starts getting involved. They take a look at it and - with a level of tact and diplomacy dependent on the begetter's seniority - inform them that the best thing to do would be to dump it and start again, but, of course, IT don't have the resources to rewrite a system which might have become business-critical very rapidly but which they didn't even know existed until yesterday.

If you are a professional software developer there is a good chance this is all starting to sound familiar. I'll round off with a list of Daily WTF style situations I have come across caused by people determined to prove the old adage "a little knowledge is a dangerous thing".

Every day around mid-morning the system (SQL Server, about 50 users) slows to a crawl for 10 or 20 minutes. Finally it just crashes. Culprit: someone in accounts has figured out how to attach tables in Access using ODBC, and then proceeds to query the two largest ones for a daily report. With a Cartesian join.

Some guy I don't know from another floor: I've put together this Access database but have got a bit stuck, could you look at it for me.
Me: Stifled groan.
Me an hour later: Why does this code export data from one table to a csv file and then import it to another table?
Some guy I now know from another floor: Because I could figure out how to export and import csv files but not how to copy data from one table to another.
Me: Stifled groan again.

Me to esteemed colleague: If you want me to export data into your (home-made) system you'll have to add an integer column for the ID.
Esteemed colleague a bit later on: I've added that column. It's a 200-character string, is that long enough?

Email from client: I am an experienced and competent VB developer but need a bit of help doing (XYZ).
Reply from me: You just need to write a couple of lines of code to do (XYZ).
Reply from experienced and competent VB developer: I don't want to get involved in anything advanced like writing code.