Enterprise Information and SOA

Almost every Information Technology implementation and IT Architecture includes Information and Data. SOA is not an exception.

The following sections will highlight some important aspects relating SOA to Enterprise Information.

SOA and Enterprise Information Integration (EII)

I recently read an Aberdeen Group Survey published in February 2008. The survey title is Enterprise Information Integration: The Foundation for Business Success and Transformation.

While reading it I almost thought I am reading about SOA:

• Abstraction

EII is an integration architecture abstracting the data source.

• Agility and Business Responsiveness

This Abstraction Layer is Business Responsiveness and IT Agility enabler.

• Black Box - Data Consumers are not aware of the information sources, Information infrastructure (e.g. databases) and location (e.g. specific instance of a database or a specific file) could be changed without changing the consumer interface.

• EII federated data approach is replacing a Monolithic data model based on consolidating data relating to an application in a dedicated database or set of tables (the Monolithic approach analogy is not perfect usually data is less monolithic than application functionality).

• Reuse

One of EII's implementation goals is increasing Data Reuse ratio, very similar to the concept of Service Reuse in SOA.

• Messaging

Messaging or data transfer is a core mechanism.

It should be noted that there is no free lunch: The price of Abstraction both for SOA and EII is overhead. The EII approach is gradually replacing the monolithic approach of a Consolidated Database. Due to high Data growth rate and growing data Complexity the choice is mainly between these two approaches: Consolidate data in one or few databases in order to simplify the architecture or Abstract Data Integration model. In addition to lack of Agility, disadvantages of the consolidation model could be time and cost of the consolidation process and possibly Performance issues due to database contention. Notice that these approaches are not mutual exclusive: applying EII does not necessarily imply no data consolidation. It should also be noted, that the consolidation model is restricted to structured information supported by current databases mechanisms unlike EII model which may support less structured data sources.

Services can be classified to four categories: New, Wrapped, Composite and Data. The forth service type is an overlap between EII and SOA: Data Services which are part of the SOA architecture and implemented by using EII.

SOA and Information Architecture

One of the Best Practices for SOA implementation is to begin with defining the Data Architecture. Information Services are part of the overall SOA architecture. Data sources outside the enterprise boundaries are the origin of some of the information retrieved or updated by these services.

According to a recent Gartner Research Note (Predicts 2008: Emerging Trends Force a Clearer and Deeper Focus on Enterprise Architecture) 80% of SOA design issues are Enterprise Information Architecture (EIA) issues. This finding is based on feedback from successful SOA implementations. Another prediction is that by 2010 most Enterprise Architecture teams will spend on EIA as much time as they currently spend on Technical Architecture. Architecture teams will have to address new information challenges derived from the new Data Architecture (e.g. Governance, Security etc.). Inability to address them due to lack of formal processes for Information Architecture will be the root of SOA initiatives failures.

SOA and Metadata

Metadata or Data on Data is an important element of SOA architecture and implementation. Metadata describes the components of SOA architecture and the underlying IT assets. SOA messages include both data and the meaning of the data i.e. metadata. The information in metadata is the key for Visibility of SOA abstractions.

SOA and Master Data Management (MDM)

MDM could be a prerequisite for SOA in order to preserve a reasonable level of data consistency and data quality. The Federated model behind EII requires one source for representing single information "truth". Mastering the MDM which maps and describes major data sources serves as this single source. Lack of adequate MDM implementation may result in ineffective Data Services and other SOA services (Functionality related Services also rely on information processed by them).

In summary, information and information on information plays an important part in SOA architecture design and in SOA implementation.

Do we need a Consumer Operating System?

In a keynote presentation many years ago I presented a Spiral model of IT Advancement or IT Evolution described in the illustration above.

Instead of advancing like more mature industries as described in the blue line, IT evolves in a spiral way: Going to one direction and afterwards to the opposite direction. As shown in the illustration IT advances on time axis, but due to the spiral pattern it evolves more slowly.

Recently Gartner Group's analysts Michael Silver and Neil MacDonald claimed that Windows will collapse if Microsoft will not change drastically the way it builds this Operating System. According to the analysts the operating system is too complex due to accumulation of code since old times Windows. The result is growing difficulty to maintain the code, which eventually will cause collapse of the operation system, unless Microsoft will shift its Operating Systems upgrades paradigm.

Windows was a standalone Personal Computer operating System afterwards it was also a Client Operating System. In the beginning of the 90^th the Stand alone/Client Operating System was Windows 3.1 which was evolved to Windows 95 (The main design goal was compatibility with Windows 3.1) and Windows 98. These operating systems could be described as Unstable and Insecure. The server and network operating system Windows NT was relatively stable and included some Security features (sometimes it was also used as a Client Operating System).

The strategy was changed when the next Operating System Windows 2000 was released: A common Operating System core with different editions some capabilities were included only in the Server or Enterprise type editions. Since than Microsoft's Operating Systems Reliability and Security was gradually improved, but the number of code lines was increased. Bugs frequency is in most cases directly related to number of code lines, so the number of bugs also increased and maintenance processes complexity increased as well.

My personal experience indicates that Vista Operating System is less reliable and a lot more difficult to manage than Windows XP (see previous posts Zen and the Art of MS Office Problem Determination and Microsoft Data Protection: Hone and Mobile Users Excluded ) . It should be noticed that the XP edition I used was the Professional and the Vista edition is Home Premium. Some people may argue that it is not Vista which is less secure and less reliable it is the home edition of both operating systems, which is less secured and less reliable than the corresponding professional edition.

It may be true, but that raises another question: Do we need a home consumer operating system at all? In my opinion we don't (at least not a complex one).

According to the Spiral Model of IT evolution we should return back to the ideas of the Network Computer (NC) advocated by Larry Ellison and Scott McNealy in the 1990^th, but in an improved and more mature way. Microsoft should deploy NC like or Terminal Server like model for the Consumer operating system, by returning to the separation between two lines of Operating Systems as deployed on the 1990^th but in a different way (again the Spiral Model): This time one line for Consumers and another line for Enterprises. The Consumer Operating System Reliability and Security should at list be equal to theses features for the Enterprise users.

The model of Home Consumers computing in the age of Web 2.0, SOA Mashups and beyond should be based upon the following ideas:

• Many Services that will be consumed will be located in the Web or in an Enterprise Servers connected to the Web. No need for complex Consumer Operating System and infrastructure software for consuming these services just a Browser or another simple consumer device (cellular phone, PDA etc.). Web 2.0 and beyond is a key differentiator between the future model and the old NC (I remember that one still have to download client software for Web 2.0 applications like Second Life so it is not completely mature yet).

• More advanced and available Mobile connectivity will enable Mobile users to plug into the Web or other networks. Technologies like Wi-Fi and WiMAX are not yet mature enough for ensuring easy mobile plugging anywhere, so probably for the Short Time Mobile users will need relatively complex Operating System in comparison to the Home Consumer.

• Consumers Devices will be connected to a Home Server or an Area Server (Serving few apartments or few houses) will serve as a partial backup in case of problems in Web connectivity, as a proxy and for hosting of some applications or services. Hosting of some applications and services may be necessary because of Network Performance considerations. The necessity for complex Consumer Operating System is eliminated because maintenance will be done mostly for the local server operating system.

It should be noticed that similar Enterprise Architecture change from Client based application centric deployment to Server based application centric deployment was performed by many enterprises.

The advantage of the consumer computing model briefly described here is hiding complex maintenance processes from consumers. Most home users are not skilled enough for proper maintaining their current complex Operating Systems. These systems should become a lot less complex or maintained by experts. There is no need for duplicate installations and maintenance of the same software on many home PCs located at the same apartment.

Most electricity consumers do not posses self-maintained home placed generators or telephone exchange equipment. The local servers' operating systems should also be less complex and very reliable as well because they are not located within the enterprise boundaries.

Market Surveys – SOA yet a Long Journey

I recently read EbizQ's SOA Market Pulse a global SOA survey done by the company via internet published on January 2008 and sponsored by IBM.

The survey is not a scientific survey, because the sample may be biased. However, Major SOA Trends are probably reflected. I would suspect that more refined findings are not accurate.

The most interesting findings in my opinion are:

Current State

Finding: Almost half of the participants have any SOA implementation and another third have plans to implement SOA.

My Take: SOA is Mainstream but not yet implemented in most enterprises. (Similar finding of 48% was found in the on the spot Israeli survey done in STKI Summit).

SOA Efforts

Finding: Most of the companies build "Islands of SOA": Multiple SOA initiatives with multiple teams and no centralized control. In some organizations there are ten different SOA initiatives.

My Take: Probably it is not easy for large enterprises to have a single unified SOA initiative and no Services redundancies. Fragmented efforts result could be a failure of the SOA implementation. This problem is not new and not SOA specific. I already faced this problem in many organizations, with no unified architecture and no central architectural group. Many years ago, I send a query to Giga Information Group (now part of Forrester Research) about future development environment of an organization: J2EE or .Net or both. A detailed background was included in the query. The analyst Mike Gilpin answered a question that was not asked: He recommended creating a Central Architecture Team. This answer was more important than the direct technological answer, because with no overall Architecture control the fragmented efforts will create a chaotic inefficient IT architecture.

Scope

Finding: Third of the survey participants SOA deployments are Enterprise-wide.

My Take: Does it indicate some mature SOA implementations or Early Adopters (Success Stories) or misconception of Enterprise-wide? Probably at least some of them implement successful Enterprise-wide services.

Technologies

Finding: Technologies frequently used for SOA Integration (in descending order): Web Services (SOAP, WSDL), J2EE Aplication Servers (WebSphere, WebLogic etc.), XML (REST over HTTP), Microsoft .Net, WebSphere MQ, ESB, Proprietary Integration Techniques, WebSphere Message Broker, Web-Enabled back-end systems with Host Emulation.

My Take:

• REST is used in enterprise context more than expected (by more than 50%)
• ESB is used a lot less than I would expect
• More than 20% are using proprietary techniques: Is it because of immaturity of SOA solutions or inadequate Organizational Maturity Level or a preliminary stage of SOA implementation?
• IBM is a leader in integration but by comparing usage of WebSphere MQ to the less frequently usage of WebSphere Message Broker (EAI tool) we can learn something about the limitations of EAI.

Finding: Most popular Technologies and approaches (including current usage and plans for 2008) in descending order: Application Servers, Composite Applications, Enterprise Service Bus (ESB), Service Registry and Repository, System Management, Pre-Build Frameworks.

My Take:

• Application Servers market leaders may emerge as SOA market leaders
• Composite Applications are important do not expect a simple architecture with no usage of current assets. Do not expect a standard based only solutions.
• Are pre-Build Frameworks adequate or we should expect a second generation of SOA implementations?

Applications

Finding: Customer Service, Sales and Marketing are the most frequent Service Enabled applications.

My Take:

• Customer applications are relatively good candidates for SOA enablement: in many cases relatively new and therefore implemented using relatively new technologies, Business Services boundaries are relatively clear and they may be non-transactional systems, most users are not Power End Users, who could be too sophisticated for the first phases of SOA implementations. Yankee Group's Research Notes few years ago identified the Contact Center as ideal starting point for SOA initiatives.
  

Motivation

Finding: Frequent motivations for SOA in descending order: Improve Business Agility, Integrate Systems, Reuse existing and new assets, Improve Business Processes efficiency, Reduced Development and Maintenance Costs, Improve Control and Manageability of the Business

My Take:

• Very different from STKI Israeli market survey results: EbizQ survey results most frequent motivation is Business Related all motivations in STKI survey results are IT related with Integration as the leading motivation. STKI results could be interpreted as indicating IT professionals only sample, however it could also indicate a wrong attitude to SOA implementations in many sites in Israel namely IT only SOA.

Wrap up

Drawing conclusion from a survey based on a sample which could be biased is dangerous. However, ebizQ survey indicates that SOA is a Mainstream Paradigm, which could be a leading paradigm in the future. More time is needed until it will be fully implemented in most IT organizations. The Journey is long, complex and difficult. Expect for many failed SOA initiatives.

STKI Israeli Market Survey sample is less biased it represents the Israeli IT market because high percent of the Israeli IT Enterprises participated. As no Business Users participated and SOA was one of many topics, conclusions based on it could be misleading as well. In spite of those considerations I would take the risk to conclude: the Israeli SOA market is less mature and failure rate will probably be higher.

Mainframe and the Dinosaurous Myth - Revisited

Many years ago in 1995 or 1996 the mainstream IT community view was that Mainframes are doomed and will be replaced by cheaper and more flexible Client/Server technology. Xephon, published a white paper titled: Mainframe and the Dinosaurus Myth. The thesis presented in this paper was that Mainframe is cheaper than Client/Server due to high maintenance costs of distributed Client/Server. 

The term maintenance costs includes higher support costs for Client/Server. After few years of experiencing Client/Server limitations Xephon unpopular theory was supported, at least as far as large enterprises were concerned.

More than decade after the Dinosaurus myth, one surprising finding in STKI Israeli Market Survey was related to Mainframe. According to the survey 40% of Israeli IT budgets are allocated to Mainframes. If you search for Mainframe in the conference presentation slides, you will not find any occurrence of it. This finding was mentioned by Pini Cohen during his presentation on Israeli Infrastructure Trends.

Explaining the results by relatively high software costs of Mainframes is not a good enough explanation: Mainframes still play important role in IT systems.
This is not the only quote referring to Mainframe important role in current IT systems of the

SOA and distributed systems age. In a presentation of DataDirect's Shadow RTE Mainframe Service Bus (Mainframe dedicated Enterprise Service Bus integrating it with services and consumers on other platforms) the company provides the following information:

• Greater than 60% of the world "system of record" data on mainframes

• More than 450 of Fortune 500 rely upon mainframes


• More commercial transactions processed on mainframe than any other platform

Even if someone will argue that the information is not accurate or not updated it is at least a rough estimation.

However, STKI's finding is still surprising because of the Israeli market characteristics: relatively few Large Enterprises and higher market share for Microsoft and Oracle in comparison to many other countries.

Mainframes are frequently used in the USA and the Western Europe markets in Banking, Credit Cards, Insurance, Telco, Government and Utilities sectors (I am excluding the military sector because part of the information is confidential or concealed).

Some successful SOA implementations are based on exposing Mainframe services to consumers in other platforms especially self-service usage via internet or other channels. Two interesting SOA Case Studies in the Telco sector exposing Mainframe Services are Verizon and Sprint-Nextel.

Similarly to the Banking industry in USA and Europe the Israeli banking industry relies upon Mainframe. However, Mainframes are less popular in other sectors in Israel, as described in the following bullets:

• Insurance – The only VSE/VM site in Israel decided to migrate to UNIX. There are many UNIX based sites and some AS400 Mainframe sites and no IBM's z/OS
  operating system sites


• Telco – Most companies were established few years ago, so the prevailing platforms are UNIX and Windows hosting Oracle, Microsoft BEA, IBM WebSphere based applications. The only exception is Bezeq, the largest company which was a governmental monopoly still deploying a z/OS based system. If a Postal services company will be included in the Telco market you can add another z/OS installation: The banking services of the Israeli Post company.


• Public Sector –Few IBM Mainframe installations but relatively to other countries larger base of HP's OpenVMS. Many applications are gradually replaced by SAP on Windows servers.

The Dinosaur Myth is only a myth: for large and complex enterprises the Mainframe (especially IBM z systems under z/Os operating system), will be valuable for long term due to Robustness, high Reliability, high Availability and incomparable Scalability (e.g. CICS Transactions Monitor with ten thousands or twenty thousands users).

The hardware platform innovations include dedicated processors for Java and Linux on Mainframe. We should wait and see if the new announced z10 computers family including quad-core processes, will fulfill its promises described as" Data Center in a Box" will be realized and support the Green IT trend.

During SOA implementation there are three mainframe strategies:

• Mainframe Replacement

• Mainframe Enablement – non-invasive approach for exposing Mainframe Services (usually through Web Services)

• Mainframe Rejuvenation – Leveraging Mainframe as a full participant in SOA architecture together with other platforms.

Rather than analyzing these three alternatives, I recommend reading Zapthink's White Paper The Mainframe as a First Class SOA Participant written by ZapThink's analyst Jason Bloomberg.

Notice that Mainframes are not only Service Providers they could act as Service Consumers and as Enterprise Service Bus platform.