Teroid Software

Megalith

2012-01-21T05:59:00.000-08:00

For some time now I have been working on no less than five major projects:

Database querying application
Database CRUD front end
Statistical analysis application
Data modelling application
Business Intelligence application

I was aware that there was a certain amount of overlap and was thinking of ways to reduce repetitive code, but today I had an epiphany. That's not some fancy ice cream, just the realisation that all these projects could be, and indeed should be, merged into one.

So Megalith is born. Whatever you can do with data, you'll be able to do it with Megalith.

Watch this space!

Dashboards? Reports? What's wrong with an export to Excel?

2012-01-19T04:02:00.000-08:00

One of the Big Things at the moment is the Dashboard - a screenfull of controls (graphs, instruments or whatever, the more colourful and eye-catching the better) which provide a snapshot of the company's data at a given point in time. Now these can be very useful if they are sensibly and thoughtfully designed, but often the design is more about aesthetics than practicality, resulting in a solution which is at best useless and at worst dangerous. It is easy to get into the habit of looking at the dashboard regularly but briefly and if the pretty pictures look good then the company and life in general is good. But unless the dashboard is exceptionally well thought out they could easily be hiding no end of problems.

So the solution? Like it says in the title, an export to Excel. And I think most people actually recognise this; the one thing I have heard from users more than anything else is "can we have that in a spreadsheet please?". What people want is to be in control of their own data. Over the past ten to twenty years most people who work in offices have built up a very considerable amount of expertise in using Excel, and given a chunk of raw data can manipulate it to get exactly what they need, often very quickly and intuitively.

The JavaScript Story

2012-01-19T04:01:00.000-08:00

Over the past 6 months I have been doing something that even a year ago I wouldn't have dreamed off - coding in JavaScript!

When I started I very naively assumed it was a sort of "Java for Web Pages", complete with the full complement of built-in support for encapsulation, inheritance and polymorphism, so it has been a bit of a shock to discover how limited the core JavaScript capabilities are, as well as being pretty weird from the point of view of somebody like me coming from a C/C++/C# background.

Despite that I have made some good progress using JS for its traditional purpose of manipulating the DOM, and also in AJAX. The next step, of course, is to use JS for more ambitious applications with the type of functionality found in desktop applications. This is where the limitations of JavaScript start to kick in and make life difficult. There are numerous workarounds to emulate interfaces, inheritance etc., but they are just that - workarounds. Of course you can write a library of use a ready-made one to provide such functionality but, however you do it, you are still using very fragile and inefficient extra code to simulate the capabilities provided as a core part of other languages. A lot of people believe HTML5/JavaScript applications are the future but, with the current state of JavaScript and the fragmented and rather motley collection of JS libraries and frameworks I find it hard to believe how that can happen for a very long time.

People have been saying for perhaps a decade that eventually desktop and web applications will merge, and HTML5 and the recent advances in JavaScript engines and libraries represent a huge advance in that direction, but unless there is something truly radical on the horizon then I can't see how this can happen for another decade or so.

So where does this leave my web development efforts? Well I will certainly be continuing to work on HTML5 and JavaScript to push the envelope of what web apps can do, but as for any ideas I might have had a few months ago of desktop-like plugin-free apps running in a browser - err, no, not quite yet!

No place for amateur software engineers

2012-01-16T05:04:00.000-08:00

There has been a lot floating around the meme pool recently about how more people - possibly even everyone - should learn to code. The most high-profile manifestation of this is the Mayor of New York taking an online JavaScript course.

At one time, maybe fifteen years ago, you were either a programmer or you weren't. Then people started playing with VBA in Excel. They created entire database systems in Access. They got a free copy of Visual Basic CCE on a disc taped to the front of a magazine. The free Express versions of C# and VB .NET came along. Now half the population are programmers.

I'd like to be able to say this is a good thing but, honestly, it isn't. However much modern development environments abstract away the complexity of software development, that underlying complexity is still there and is actually very much greater than ever.

To tap away or drag-and-drop away without understanding or even being aware of that complexity is just the path to chaos. There are many fundamental concepts that "proper" programmers don't even consciously think about, they are just hardwired into their brains, but they are crucial to their everyday work. But the amateur programmer, blissfully unaware of this, cannot help but create inelegant, inefficient, unfathomable and unmaintainable code, whilst being intensely proud of what is probably the most complex thing they have done in their lives.

You might argue, quite reasonably, that the sort of development done by non-professionals if of the type that would never get done at all by professional developers, and that a far from perfect system is better than none at all. This may well be, but the scope of many systems grows rapidly, and a small system put together by somebody in accounts or marketing or wherever just for their own use gains interest from colleagues who ask to use it or for it to be expanded. Somebody else asks for it to be integrated with some other system, perhaps the company's "official" system. The system's proud parent runs out of talent and then the IT department starts getting involved. They take a look at it and - with a level of tact and diplomacy dependent on the begetter's seniority - inform them that the best thing to do would be to dump it and start again, but, of course, IT don't have the resources to rewrite a system which might have become business-critical very rapidly but which they didn't even know existed until yesterday.

If you are a professional software developer there is a good chance this is all starting to sound familiar. I'll round off with a list of Daily WTF style situations I have come across caused by people determined to prove the old adage "a little knowledge is a dangerous thing".

Every day around mid-morning the system (SQL Server, about 50 users) slows to a crawl for 10 or 20 minutes. Finally it just crashes. Culprit: someone in accounts has figured out how to attach tables in Access using ODBC, and then proceeds to query the two largest ones for a daily report. With a Cartesian join.

Some guy I don't know from another floor: I've put together this Access database but have got a bit stuck, could you look at it for me.
Me: Stifled groan.
Me an hour later: Why does this code export data from one table to a csv file and then import it to another table?
Some guy I now know from another floor: Because I could figure out how to export and import csv files but not how to copy data from one table to another.
Me: Stifled groan again.

Me to esteemed colleague: If you want me to export data into your (home-made) system you'll have to add an integer column for the ID.
Esteemed colleague a bit later on: I've added that column. It's a 200-character string, is that long enough?

Email from client: I am an experienced and competent VB developer but need a bit of help doing (XYZ).
Reply from me: You just need to write a couple of lines of code to do (XYZ).
Reply from experienced and competent VB developer: I don't want to get involved in anything advanced like writing code.

Using a Test Matrix in NUnit

2011-12-14T08:20:00.000-08:00

The conventional way of using NUnit is to write a separate method for each individual test of a method.

This can very quickly get unwieldy so I put together a simple example which creates and populates a test matrix of input parameters and expected results, which is then iterated through by one single [Test] method.

Firstly, we need a method to test, and this is it. As it is just for illustration I have kept it as simple as possible.

public class Methods
{
    public int Add(int a, int b)
    {
        return a + b;
    }
}

Then, within the unit tests project, we need a class with properties for the input parameters and expected result. It also has a property which is basically the other properties combined into a human-friendly string.

public class AddParameters
{
       public int a { get; set; }
       public int b { get; set; }
       public int ExpectedResult { get; set; }

       public string ParametersAsString
       {
            get
            {
                 return "a: " + a.ToString() + ", b: " + b.ToString() + ", ExpectedResult: " + ExpectedResult.ToString();
            }
       }
}

The next step is to create a collection of AddParameters objects. This collection forms the test matrix itself. Giving the method the [SetUp] attribute forces NUnit to run it before running the tests.

List AddTestMatrix;
     
[SetUp]
public void CreateTestMatrix()
{
    AddTestMatrix = new List();

    AddTestMatrix.Add(new AddParameters { a = 1, b = 3, ExpectedResult = 4 });
    AddTestMatrix.Add(new AddParameters { a = 3, b = 5, ExpectedResult = 8 });
    AddTestMatrix.Add(new AddParameters { a = 9, b = -3, ExpectedResult = 6 });
    AddTestMatrix.Add(new AddParameters { a = 22, b = 9, ExpectedResult = 31 });
    AddTestMatrix.Add(new AddParameters { a = -77, b = 78, ExpectedResult = 1 });
}

Maintaining the matrix is as simple as editing this method.

The final step is to write a method to actually run the tests:

[Test]
public void TestAddUsingMatrix()
{
    Methods methods = new Methods();

    foreach (AddParameters ap in AddTestMatrix)
    {
        int Result = methods.Add(ap.a, ap.b);

        Assert.That(Result, Is.EqualTo(ap.ExpectedResult), ap.ParametersAsString);
    }
}

How difficult is it to design a database?

2011-11-30T07:20:00.000-08:00

Well actually, it is not that difficult at all. There is a straightforward set of procedures which,
combined with a bit of common sense and experience, almost form a foolproof recipe.

1 Identify the entities of your system (products, customers, orders etc). Each of these forms a table in your database.
2 Identify the attributes of each entity, and the type of data each represents - Name, Address,
Description, Price etc. These form the columns and column types in the database.
3 Identify any existing unique identifiers for individual instances of entities, eg customer numbers, products codes etc. These form the primary key of the table. If there are't any, create them.
4 Work out how the entities are related, eg one customer can place many orders, one order can contain many products. These form the relationships in the database, implemented as foreign key constraints.

If you follow this recipe properly then your database will automatically be pretty well normalised at least to 3NF or BCNF, or at least any deviations from this will be minor and easily rectified. The design might not be great but will be a lot better than the majority of databases out there.

Obviously I have massively simplified the process that thousand-page books have been written on, but, in essence, that is it. But despite that many very good software developers who can write superb code end up cobbling together appalingly bad database schemas. The most common problems are

1 Repeating columns ("Phone1", "Phone2"). This causes major problems in querying and updating, and what if somebody has 3 phone numbers?
2 The wrong column types, such as as char type for a number. You then have to scatter your SQL and application code with conversions back and forth, and run the risk of somebody entering Fred as a price.
3 No primary keys. No primary keys = no foreign keys. No foreign keys = no relationships. No
relationships = no truly relational database, no data integrity, no reliable and efficient queries.
4 No unique constraints. Some widget or other gets entered twice, chaos and bafflement all around.
5 No domain tables. Somebody enters all their American transaction with currency USD. Somebody else tends to prefer US$. A third person comes along to add up all the USD transactions and only gets half of them. You don't need to be an accountant to see the disadvantages of this!

The principles of relational databases and their design have been around since the early 1970s. That's almost prehistoric in computer terms, but they have stood the test of time and I see no reason why they shouldn't last another 40 years. And that is because they work, when properly implemented. They give us data integrity and fast, efficient and straightforward inserts, updates, deletes and queries, as well as forming a solid foundation for creating denormalised data warehouses when needed.

But far too often they are not properly implemented, not because it is too hard or considered unimportant, but because the expert C#, Java or PHP developer is hardly aware of their existence, and nothing in their training or experience has led them to believe that database design is anything more than something you just get over and done with quickly so you can get on with the real task of writing your application.

Of course, in an ideal world the software architect or project manager would appoint a database designer to do the job properly before a single line of application code is written. But this is the real world and application developers who really only know how to develop applications are expected to do everything including databases design.

Why should you Blog?

2011-11-18T10:50:00.000-08:00

If you happen to be an archeologist you probably like to classify human history in terms of what the stuff you dig up is made of - stone age, bronze age, iron age, Coke can age etc. This is all very well but only gives an extremely limited and even misleading picture of the development of mankind - for example we use more iron now (in the form of steel) than they ever did in the Iron Age.

I think a much more accurate and profound way of categorising the epochs of human history would be to use the technologies used to store and pass on thoughts, ideas and facts. They could be something like:

The Gestures and Grunts Age
The Speech Age
The Handwritten Text Age
The Printed Text Age
The Internet Age

What defines us as humans isn't what we do, but how we teach and learn, each generation benefiting from the accumulated knowledge of our ancestors. Each major advance in the technologies used to do this represents an increase of one, two, perhaps three orders of magnitude in our capabilities.

A central thread of philosophy since Classical times has been, how do we know what we know, how do we learn facts, how do we find out new information? The Ancient Greeks believed we were born knowing everything, so to gain knowledge and wisdom one just had to sit back and think. (Whether they really believed this, or just used it as an excuse to sit around doing nothing I couldn't say).

The more Enlightened philosophers such as John Locke believed the mind was a blank slate, formed and shaped by each person's unique combinations of experiences. This is obviously much closer to the truth, but the mechanisms involved in turning the simple firing of synapses into a mind capable of original thought remain a mystery. One thing is, I believe, clear though - pour enough knowledge and experience into a human brain and it will chop them up, mix them, and stick them back together in ways which, in just a few thousand years, has transformed us from hunter-gatherers with a lifestyle little different from mere beasts, to members of the incredibly complex society of today.

A couple of decades ago, only a small elite of writers and journalists enjoyed the privilege of having their knowledge, thoughts and ideas added to the melting pot of human knowledge. Now everyone can - it's not a privilege any more. But should you? Is it worth it? There is so much text sloshing round in blogs and elsewhere that most of it is hardly read. A few people might read a given post in full, more might skim it, the vast majority will remain ignorant of its existence. But it still serves a purpose if just a single person reads or even skims it and picks up one or two thoughts to add to their own personal mindset. Even if nobody reads it, maybe it benefits the writer by helping to clarify the jumble of thoughts in their own mind.

But what of the future, the Sixth Age? Well that, IMHO, will be when computers go beyond just storing text and delivering it to people to read, and start reading it themselves. And not just reading it, but chopping it up, mixing it, and recombining it into something original just as our brains do.

I am not going to get into futurology and try to predict the consequences of all that, or whether some HAL-like supercomputer will take over the world. I'll just say that what you think, believe, imagine and then type will in some small way form part of the inherited wisdom of mankind.

So that is why you should blog.

The First Post

2011-10-24T01:51:00.000-07:00

It is now ten years since Teroid Software was formed and, at long last, I have a blog. Aren't blogs great: you can free the writer in you without the bother of having to write a whole book or having to get past some dimwit publisher who is too stupid to recognise your literary genius :).

Around the time the company was formed there was a big upheaval in Windows software development, caused by the launch of the .NET Framework and C#. There was a lot of confusion and scepticism about the whole thing - confusion because people didn't understand what the .NET Framework was all about, and scepticism because C# was widely regarded as just a rather cynical attempt by Microsoft to rip off Java.

Many people - me included - decided not to jump on the .NET / C# bandwagon immediately, but just to wait and see how it panned out. Just because something is from Microsoft doesn't mean it will be a success (anybody remember Bob?). Incidentally, some people waited a very long time to see how it panned out: it was only a couple of years ago that somebody told me they were starting a new VB6 project!

Well of course .NET and C# panned out quite well quite quickly, and a couple of years later I gave up Visual C++ and started working on my first serious C# project: a Venn Diagram control for WinForms.

Since then I have written a lot more WinForms controls, a few WinForms apps, a few WebForms controls, a couple of WPF apps and a handful of Silverlight apps, all in C#. (I have also done a bit of Java and PHP, but please don't tell anyone about my shameful secrets.)

And now, it seems, we are approaching another major upheaval at least as big as that brought on by .NET and C#, and probably one causing even more confusion. People have been saying for many years that desktop and web applications will one day merge, and it seems this goal is at last in reach. Silverlight - at birth a minor offshoot of WPF - has taken off in a far bigger way than Microsoft or anybody else could have imagined, and in parallel there has been a big increase in interest in HTML5/JavaScript RIAs. Which of these technologies, Silverlight and its XAML successors or plugin-free HTML5 and JavaSript, will dominate I wouldn't like to speculate. Maybe there is room for both - HTML5 Rich Internet Applications and XAML Richer Internet Applications.

I am currently being prudent, sitting on the fence, hedging my bets, being indecisive - take your pick. I am fully committed to the RIA as the way ahead but with one foot in Silverlight and one foot in HTML5/JavaScript. Time will tell.

And what of WPF, at one time Microsoft's Next Big Thing. Well it's not very big is it? It took a couple of years to even show up on the radar and has now been totally eclipsed by it's little sister Silverlight. Why? Well, it was out of date from the start in that, conceptually, it was no more than the next WinForms at a time when people were thinking much wider. Even people who only need a desktop app often don't bother with WPF, seeing no reason to break away from their familiar WinForms. Fair enough!

A big problem I have always had with software development is having more ideas than I can ever have time to implement. Half of the ideas I dream up never get started, and half the ones that do get started end up being prudently triaged to free up time to actually get something finished. I have whittled down my current projects to just two RIAs - Pixentient photo gallery, and Inferno statistical analysis platform. More on these in my next posts.