Open Protocols, Open Formats

Definitions

The obvious start would be explaining the title of this post, that is, the meaning of “open protocols” and “open formats”. In particular, I will define “protocols” and “formats” to give a common basis for this discussion.

A “protocol”, in computer terms, is a communication language allowing two or more pieces of software or computers to send data between themselves to carry out some task. For instance, computers communicating over the Internet follow the Internet Protocol (IP), which helps determine how data gets from a start point to an end point, among other things. Also, when a computer makes a request for a HyperText document (an HTML webpage), it uses the HyperText Transfer Protocol (HTTP) to communicate this request to the computer hosting the document, and to understand the response.

A “format”, in computer terms, is a language for recording information in a computer-readable way. For instance, when you save a plain text document, the computer will usually use the ASCII format, which specifies a conversion between letters and code numbers. These numbers are then written in binary and stored on some physical medium. Also, when a computer has to store a picture, there are formats like PNG which specify how to split the picture up into blocks of colour, and how to assign numbers to those colours in an efficient way so that the whole picture can be stored as a sequence of binary digits (“bits”).

Simply, then, an “open protocol” is a protocol for which the specification describing it is freely available, and an “open format” is a format for which the specification describing it is freely available. What does “free” mean? At best, it means the specification is a document that can be read, distributed and used without restriction, especially without putting any additional restriction on software someone might write which follows (“implements”) it.

Successes

I have already mentioned some examples of successful formats and protocols: IP, HTTP, ASCII and PNG. In fact, most of the Internet and the Web is built on open protocols and formats, which describe everything from the addresses (“URLs”) used to access resources, to the HTML family of formats, one of which this page is written in (view the source of this page to see what that looks like).

These open protocols and formats, and others like them, are the cornerstone for every computer, enabling the multitude of capabilities modern devices provide. They stretch far beyond the mundane personal computer, into servers, mobile phones, wireless routers, cars, medical practices, weapon systems, and space missions. Since the beginning of computers, open protocols and formats have been with us allowing us to build almost everything we have today, and if we support them we will reap the benefit long into the future.

Justification of successes

It shouldn’t be too hard to see why openness has been a benefit. A basic principle of science is to “stand on the shoulders of giants”, the antithesis of “reinventing the wheel”. Whenever an engineer has been faced with creating something new, something which enriches society and rewards themself, it has been an obvious option to build it using the well-established, tested and understood open protocols and formats which are available to them and to anyone. The skills and accumulated knowledge of the other people who have used these building blocks is then available to them, accelerating their own work and allowing them to concentrate their efforts on the truely innovative part of their invention.

Often, the same logic works to encourage the engineer to make their own creation open in terms of its protocols and formats (while reserving the right to keep the actual software of their specific implementation protected by copyright). By being open, third parties can build compatible products and services around their creation, making the original creation a more valuable proposition to potential customers. The skills of the customers themselves should also be more transferable to the new product if it shares a basis with common existing products.

For example, if someone was to create a 3D animation program, they would be sensible to include support for importing and exporting models in an open format, so that customers could use their favourite 3D modelling program to create models for it, and manipulate scenes it generates. Customers could even write their own modelling program, or pay a company to produce one for them, designed to work in harmony with the 3D animation program. This makes the program accessible to more people, and allows them to get more out of it, so they would choose that product over a competitor’s.

Dangers of foregoing openness

From the above it should be clear that openness at least has some merit. In fact it should show that in many cases, openness is the main criteria to judge a technological decision on, but I understand that the engineering benefits may not seem like immediate practical advantages to a consumer. I will thus have to explain the above in terms of the hidden costs of choosing an alternative to an open protocol or format, which should go some way towards making those alternatives less attractive.

• Firstly, there is the not-so-hidden cost of the extra engineering work which has to go into something that reinvents the wheel. It is of course the end user that has to pick up the bill for all these wasted hours, so even if you don’t pay a seemingly large cost up front, you probably will in the long run. For instance, the business model of the company might be to sell you a product at a loss, and overcharge you on services for it, to recoup their inflated development cost.
• Similarly there is the time cost, since reinventing the wheel also takes longer than reusing existing, known-working solutions. Again, you may not notice the time taken if the product being offered is the first to market, but you can expect the time between updates to be slower, and thus future innovation to happen more slowly. In fact, cost, time and quality are all related, so you might similarly find that the product, or any updates or subsequent versions, are faulty because they’ve been rushed or under-funded.
• Next, there is the problem that the closed protocol or format used requires you to abandon previous knowledge and learn a whole new system. For instance, a new closed protocol might require you to use a different naming system for your PCs or documents, or require some peculiar configuration settings which were not needed before. A new closed format might require you to re-save all your old documents in the new format, or do a batch conversion, which may lose information in the process. Of course, new open protocols and formats can have the same problem, but the fact they are more openly documented makes it easier for new software to support old versions.
• Surely the most obvious consequence of choosing a closed protocol or format is the problem of vendor lock-in. If one company or group control a secretive protocol or format, then you do not truely own your data any more. You have to use their products to communicate or read your data, because only they know the secret which locks it up. Companies understand this very well, which is why openness is under such a threat. Once some closed product or service has a large enough user base, it becomes a cash cow for a company, as they can force their customers to “upgrade” (to versions which fix flaws that should never have existed in the original in the first place) rather than buying alternatives and can name their price, knowing that you will pay anything to keep your data working.
• Carrying on from this, there is also the perverse incentive for companies who achieve vendor lock-in to make their protocols and formats not backwards compatible, that is, to make old versions of the product no longer work when newer versions come out. This keeps customers on the upgrade treadmill, and often requires customers to purchase new hardware to support the new unneeded features in the latest version. As mentioned earlier, the switch to using the “upgraded” version can be unnecessarily troublesome and costly in itself.
• Finally, there are the standard complaints that a large company with a monopoly doesn’t even have to try, can afford to be negligent about security and not face competitive pressure to innovate. Even worse, you are dependent on that company staying in business and continuing to support the product line. If the company goes bust, or gets bought out by a company with a different focus, or gets sued for patent infringement in the product, or just thinks that there isn’t a big enough market to continue working on it, then you may not even be able to buy software from that one company.

Proof

There may still be sceptics who would say the above is just philosophical posturing, not relevant to the real world. Also, the above may not be easy to remember when it comes to actually making a choice in a practical situation, as humans tend to remember quotes and anecdotes better than logical arguments. I therefore include some recent examples, showing the disaster scenarios I put forward are not merely theoretical.

• Connection cables, like the ones between a mobile phone and a PC, are probably something most people have had experience with. While not a “format” or a “protocol” in the software sense, their design can be open or closed, and the issues involved apply here too. Companies are well known for unnecessarily creating new types of connector for the ends of their cables, so that if the cable breaks they can charge you an extortionate price for a replacement. Taking this practise to the extreme are companies like Microsoft (who recently said that they will take a cut of all profits made from devices which connect to the XBox 360) and Apple (who require that third party manufacturers of accessories for the iPod to pay for the right to connect to the device).
• Apple deserve another dishonourable mention for their iPod and iTunes hegemony. By using the closed system around the AAC music format, they make it almost impossible to put music from iTunes onto any device other than an iPod. This means people who want to use iTunes are forced to go out and buy an iPod, even if they already have a better, cheaper portable music player. In order to defend the restrictions which stop the format being converted to something more compatible, Apple even threatened with legal action the hosting provider of a software project designed to enable this fair use format conversion. Conversely, if someone chooses to buy an iPod, they have little choice but to use the iTunes software and service. When a company tried to make their online music store compatible with the iPod, Apple also made legal threats and warned users that they might reprogram the iPod to make songs from this alternate source unplayable.
• Another current fad, the BlackBerry mobile messaging device, has also clearly shown the danger of closed protocols, in a case of poetic justice. As I understand it from the limited information the public know about the product, it uses a closed protocol to communicate back to a central server, controlled by the company. For at least some users, then, all your messages are stored on a server which you don’t have access to, and you cannot choose to use a server run by another company, offering a cheaper monthly plan, say. Similarly you cannot choose to download all your emails from that server with your favourite desktop email client. This means your entire message correspondence with people could be available to that company, and exist only on their servers. Almost to prove the stupidity of this model, the company is now having to face the prospect of ending the service in the United States due to alleged patent violations of the product. So a company which made money from blocking customers from using competitors has had their customers blocked from using their service by a competitor.
• Finally for this list, although there are more examples being created all the time, consider the case of the Norwegian public broadcaster putting their video clips and radio stations online in a Microsoft closed format. This means that any company that wants to make software to access Norway’s public resource needs to pay Microsoft for permission to compete with them. Even worse, the continued existence of this cultural archive is dependent on Microsoft’s good will or the profitability of them supporting it. As one Microsoft employee wrote “Microsoft could even shut itself down and take it’s [sic] video codec ball home and stop everyone from playing.”. Many a true word spoken in jest, as this admits the power Microsoft hold over their customers.

Conclusion

Even if you don’t think you understand the definitions, or you think you’re just a casual buyer that doesn’t need to know the engineering issues involved, this does still affect you. There is a simple test of openness: check if there is a compatible competing product. If you are buying a product that creates files, can those files be read by a competitor’s program? If you are buying a product that communicates with other products, do those products have to be made by the same vendor? If you are buying into a service, can third parties add features to that service? If you use a service that creates and stores data, does it let you download and migrate that data?

Asking these simple questions now will save hours of frustration and countless expense later. By not protecting your interests, you are letting a company do whatever they want, whether it’s starting to require money for a service that used to be free, scrapping support for a buggy program you rely on, adding unnecessary features which slow the product down, refusing to keep up with competing products, including spyware, taking away existing features or rights you had, or some other money saving / money making scheme a company has yet to invent.

Protocols and formats of all software open up! You have nothing to lose but your chains.