The Limits of Open Source - Vertical Markets Present Special Obstacles

For those who may not know, vertical market software is software that is designed to service the specific needs of one industry. In its ultimate form, it wraps all of a single business's or organization's application needs into one comprehensive package that covers everything and relates all the data so nothing needs to be entered twice. This is the holy grail of business computing and can rarely be achieved in large businesses but in the smaller businesses that much vertical market software is aimed at, it may be acheivable.

This page and the following page on medical practice management software are focused primarily on the comprehensive vertical market packages that get names similar to Industry Name Management System. Generally included under the term vertical market software are much more specialized products that are designed to serve a single function or closely related group of functions within the target industry. This discussion is not aimed at these products as there are already many examples of open source vertical market products of this type.

Some well established vertical markets include law office practice management, the non profit industry, doctors' office management software, schools, and real estate sales. Vertical markets normally have IT staff but typically the professionals or managers do not have technical (programming) skills that lend themselves to open source projects.

Even an area that seems as specific as law offices is served by an array of different and overlapping products. A large category is legal time and billing systems; most law firms use these but a firm that specializes only in tort cases, and is paid solely as a percentage of any judgements which are awarded, doesn't need such software. Most law firms need document management software. Some will immediately think that is a type of horizontal market software used by many industries. This is true, but because of law firms' highly specialized needs to relate documents to specific cases (normally called matters), sources, witnesses, annotate those documents, search the annotations, and other specialized needs related to document management, general purpose document management systems are rarely adequate. A few cases have been so large, that they required custom document management systems because of the massive number of documents involved, and the need to be securely available to multiple plaintiff and defendant firms.

In the non profit area, organizations first split between member based organizations and donor based organizations. Member based organizations typically divide into professional and trade associations. There are donor organizations that look like member organizations because they call their contributions, dues. A national organization may have subsidiary regional or local chapters or may be a federation of such. The regional and local organizations may be entirely independent and pursue similar goals but simultaneously be the national's prime competitors regarding dues and donations. Where there is a real affiliation between the national and regional or local, dues may be collected nationally and in part passed on to the affiliates or may be collected locally with a share going to the national or they may even be collected at either level and split regardless of which organization is the recipient. A single organization may have over fifty dues classifications (not counting local and regional) based on various combinations of occupation, location, time in the profession, and specialty areas the member joins.

Ideal vertical market software would capture in source code (for speed and simplicity) everything that is common to the entire market and have configurable tables (for flexibility) for everything that varies from one company or organization to the next. In practice this is almost impossible to achieve because it would require a comprehensive knowledge of the industry and even industry experts would rarely posses this level of knowledge let alone this knowledge plus the design skills to translate that knowledge into a workable system design that could be coded. I've seen great utilities and excellent horizontal market applications. My exposure is quite non random but I sense few users of vertical market software are highly satisfied.

Someone recently asked me, wouldn't a management system for a doctor's office and a lawyer's office have much in common? The answer is an unequivocal no. At the very lowest level, just above the programming language, they might share some basic infrastructure such as application menu, security, and logging features. These and a few other low level functions are about as far as an open source approach could usefully share between different vertical markets. Low level infrastructure features tend to be the types of functions that are most poorly done in vertical market software because clients don't ask for them. Developers concentrate on visible functions requested by clients. This is despite the fact that good developers know that time spent up front to do the basic system architecture well, pays for itself in the long run. This becomes apparent in the latter stages of application development, once the client starts requesting changes, and even more after working systems are delivered.

Vertical market software problems come from who builds them and why? Most vertical markets are too small to be of any real interest to the large software manufacturers who create software in the hope of selling millions or at least hundreds of thousands of copies.

In contrast, vertical market applications tend to be sold to a handful to several thousand customers. Established vertical market companies, may have a few dozen to a few hundred employees, but are typically started by a few technical staff, who have worked in the field they hope to build software for, and who come together to form a company. Typically, they are too small to even attract venture capital until they have been around for a while and achieved some measure of success.

As the owners' savings are typically riding on the outcome, they must make profits quickly. This means that they need to deliver working systems to clients on the first iteration. As a result, the first systems delivered are really custom systems. As clients are added, functions are generalized and a "package" emerges. Smart companies that are doing well, will take their combined knowledge from several clients and do a new system from scratch that will allow most configuration changes to be made in tables with few in source code. Still, if things continue go well, new clients will push the limits of the second generation system.

Often three tries are needed before a vertical market company really achieves a solid package. The first client's pay for the vendor's learning mistakes. This learning curve typically aids a handful of companies that dominate any specific vertical market and makes it hard for a new company break in. Employees of existing companies are typically bound by non disclosure and non compete agreements making those who could best start over and get things right, not available. New entrants are often IT workers from the affected industry, and who have been on the buying side. They start new companies with detailed knowledge of the companies they came from but not necessarily broad knowledge of the industry. Besides repeating beginners' mistakes, the new companies face established competition.

Vertical market software companies don't have the resources to develop really comprehensive packages from the start. The result is a mixture of specialty packages that address a specific need or related group of needs. They tend to do a few things very well but don't integrate with other software. These are directly comparable to the kinds of vertical market software where there is already significant open source representation if not necessarily adoption.

Some low end packages attempt to be comprehensive or nearly so. They will have limited customization capabilities, forcing businesses to their way of working and do a variety of things, but not really do any of them very well. As products mature, they get better as more customers ask for more features and configuration options but their price also escalates as the functionality increases. Unless companies deliberately build new systems from scratch, and even then depending on the breadth of vision applied to the new products, poor design decisions made in product's early versions, hamper future developments. The result is that vertical market software tends to fulfill specialized needs well, generalized needs at a modest cost but poorly or comprehensive needs moderately well but at high costs. The small client can rarely afford good, comprehensive software, if any is even available.

Proprietary vertical market companies, typically started by a two or a few individuals who are putting their life savings and all their time into the product to achieve that high level of initial functionally vertical market products typically require, often have to struggle. The motivation is clear; if they are successful, the payoff can be quite large and the creators reap the rewards.

Most of the successful open source projects either started as a small tool or experiment for the author's own benefit (Linux, Perl, PHP) or used a preexisting base (*BSD, Apache) as a starting point to achieve a minimal level of functionality. The useful starting point for a comprehensive vertical market management system is likely to require much more up front work and the beneficiary (target of the software product) will in nearly all cases be a non technical customer, not the author of the software. I won't say it can't happen but what is the mechanism and incentive to pull together and hold together long enough, a group of people with the resources and knowledge to make this complex product, which is then given away?

Proprietary vertical market software may or may not provide source code to customers. The more package like the product, the less likely the customer has any access to the source. Where vertical market applications aimed at large customers, are extensively customized at the source code level, the customer often does have access to the source code. If vertical market source is available at all, it will nearly always be with highly restrictive non disclosure agreements. Either way, the vendor is for most practical and likely legal purposes, the only company available to support the vertical market application. The customer typically has three choices, pay the vendor's maintenance fees regardless of how poor the service may be, attempt to use the software without support which is generally not a viable long term solution, or find another product and vendor and hope that relationship is better.

Open source insures that anyone who can program in the language, has an opportunity to maintain the software. Larger and more complex the applications give the developer an initial but not insurmountable advantage. If access to the source is unrestricted, any company that can provide the best maintenance value, is over time, likely to win as much of the maintenance load as they can service. Given today's online connectivity, geographical location is much less important than it once was. No user of open source applications, is ever tied to a single maintenance provider.

In few areas except desktop operating systems, are customer choices typically more limited than in comprehensive vertical market software. There may initially appear to be many products, as in the legal field, but even a preliminary investigation is likely to show the number of comprehensive products suitable for a particular practice is small. If the choices are limited before a customer makes a choice, once a customer makes a choice, the business model is even more unfavorable to customer. There are few areas where open source would provide greater benefits to the users, if the obstacles to building an initial viable product and getting it used sufficiently widely could be overcome.

There are already many open source vertical market products but they are generally narrow function products. They were not designed to and do not integrate well to form a comprehensive management suite. I've been in the postilion of managing a modest size business's technology with a handful of custom applications, some of which copied data from a "central" database (but afterwards did not synchronize data updates) and others which maintained their data entirely independently of other applications. A business with multiple pools of overlapping, uncoordinated and inconsistent data, can't do useful analysis and can't embark on significant new IT initiatives, until core business data is centralized (or at least effectively coordinated).

This is the one thing that comprehensive proprietary vertical market products typically do with some degree of success: centralize the data. Almost any centralized system is better for the business or organization than a collection of uncoordinated applications regardless of how poor any individual user or department thinks their function or module is in the centralized system. If an application updates data that has come from a central repository, the updates must be returned to the central repository without overwriting any other updates made in the central repository. As much as any user or department may like an application, if it creates multiple inconsistent collections of the same data, it will cause more harm to the organization in the long run than any benefits perceived by the application users.

I do not claim to have made an extensive search. The searches that I have made, have yet to turn up a single widely used comprehensive vertical market management application. This is application that performs enough functions of the target business, that it's reasonable to say that the business is managed through the application. For these purposes I'll accept 10 real users (not testers or evaluators) as widely used. For those projects that are attempting to be comprehensive, none appear close to the goal.

Even more important than comprehensive functionality is that the database be well designed so that it can be extended to areas that are not currently covered without changing existing data structures.

For example, if an open source project to run a small family medical practice actually achieved real use, then it might form the basis to be extended to other practice areas. Then it might be extended to larger practices. This is classic open source incremental growth.

The need for an initial high level of functionality and fields made up of professionals and decision makers lacking technical skills, were previously mentioned. There is another issue that differentiates most comprehensive vertical market management applications from the areas where open source has proven successful. Successful open source projects are typically process driven where comprehensive vertical market projects are business related and thus data driven.

For anyone who has not worked extensively with business data base applications, it may be hard to appreciate the difference. Procedural thinking, building custom applications to support specific business processes with their own data, is what got many businesses into major IT trouble in the 70's and 80's. It's virtually impossible to mesh such applications into any useful whole. Design a database that accurately reflects the entities that a business works with, and putting the right application processes (programs) over them to adapt to current and changing business needs is comparatively simple.

Designing a database to match a business is hard. Designing a database to match a very specific industry is considerably harder because of variation from business to business. Designing a database to account for a broad range of loosely related industries, like non profits or law firms becomes an enormous challenge. By comparison, configuring a process is easy. Some permanent data structure holds the options. If 1 do a, if 2 do b, if 3 do c, etc. It really doesn't matter how many options there are or how they interrelate, you can build a data structure and matching logical structure to handle any level of complexity.

Relational databases are somewhat different. For a single entity (table) you can have as many empty fields (representing optional attributes that may or may not be used) as you want, but in any non trivial environment normalization issues become important almost immediately. That is, how do you deal with data that might repeat? Relational design requires this be placed in separate tables. It's not necessarily obvious if data repeats or not. Are home and work address equivalent entities that should be placed in a separate table or are they distinct attributes of an individual? Should individuals and companies be treated as similar or do they require different tables? The answer to this depends on how they are used in a business. What if different businesses in the same industry use them differently?

Are two addresses sufficient? In some industries yes but not in legal associations. Many lawyers maintain offices is two or three states (typically near borders and in "tri-state" areas). Some have summer and winter homes. Periodicals should go to the seasonal residence, bills to the primary office and all three should be listed in the directory. One answer is that the computer system can't handle that many addresses. If the member contributes $10,000 or $100,000 per year, that may not be an optimal answer. If the system can't handle this, staff will try to find some manual method or system "workaround" that allows it.

One solution is to separate all addresses from individuals and companies into an address file and then relate addresses to individual and businesses with files containing primary key pairs and perhaps codes defining the relationship. This makes every program that uses this data more complex. How many individuals can have the same address? What if one of them changes it? Tie the address to the company and the company to the individual. This is a new, different and still more complex relationship. The point for anyone who has not done database programming is that changing these kinds of fundamental data relationships tends to affect nearly every program in a system. These aren't soluble with simple global search and replace operations either. Changing the relationship between tables changes the logic of programs that work with the data. Every program that touches the data has to be reexamined and fixed in some way.

It's hard to come up with any plausible analogy that people used to dealing with system programs will understand but I can think of an implausible one. Assume that on the next Jan. 1, the format of the IP header was going to change. Forget all the practical issues and suspend disbelief for a moment. The specific change doesn't matter. Every program that uses the IP header assumes it's current format. Every program that uses the IP header, would need to be examined. Depending on the change some programs might not even need fixing but others might require major revisions. If the change involved the removal of some piece that was important to a particular program, there might be no satisfactory resolution. Everything about the protocol assumes the structure of the header and even a minor change would create enormous problems. That's why we don't change an established protocol like IP. Instead a new IPv6 is developed with enough compatibility that the two can work together and a phased replacement is planned.

It may seem hard to believe, but business database application systems become just as dependent on the core table relationships and certain keys as the entire IP protocol and all the programs that use it do on the IP header. Modern databases and applications are very flexible about adding new fields and tables. And any database or application is likely to have infrequently referenced tables. A search for all the programs that reference a specific table might show it can be discarded or replaced with little consequence. At the same time any applications of any significance, and any comprehensive vertical market application will be significant, will have some core tables and relationships on which everything else rests. Change these and you might as well start from scratch. Early in the design process there may be two or more fundamentally different approaches that are seriously considered. It's highly likely that one will fit with the way one business works better than another. The alternative designs are very likely to work better for some other business. A choice must be made and hopefully its the one that works best for most businesses in the target market. Beyond a certain development point, it makes no difference whether the right or wrong choice was made, it will never be changed.

I expect there are a variety of projects grappling with related issues (not necessarily addresses) but personally I haven't seen examples that appear to have solved them. I'll discuss some examples on the next page on medical practice management software. In today's age of personalized systems, failure to meet the customer's, member's, or contributor's desires often results in a loss of their business. Proprietary vertical market software does not do a very good job, and surely not at cost effective prices. A big plus for open source is that it's not driven by the profit motive but it has other obstacles to providing high quality, comprehensive, vertical market software

Top of Page - Site Map

Copyright © 2000 - 2014 by George Shaffer. This material may be distributed only subject to the terms and conditions set forth in http://GeodSoft.com/terms.htm (or http://GeodSoft.com/cgi-bin/terms.pl). These terms are subject to change. Distribution is subject to the current terms, or at the choice of the distributor, those in an earlier, digitally signed electronic copy of http://GeodSoft.com/terms.htm (or cgi-bin/terms.pl) from the time of the distribution. Distribution of substantively modified versions of GeodSoft content is prohibited without the explicit written permission of George Shaffer. Distribution of the work or derivatives of the work, in whole or in part, for commercial purposes is prohibited unless prior written permission is obtained from George Shaffer. Distribution in accordance with these terms, for unrestricted and uncompensated public access, non profit, or internal company use is allowed.