Eric Newcomer's Weblog

Conference Season - Upcoming Speaking Engagements

Wed, 30 Apr 2008 17:04:56 -0500

The snow is finally gone and the new, yellow-green leaves are starting to emerge... Yes, that's right, it's conference season again!

Here's a list of conferences at which I'll be speaking during the next couple of months, followed by a summary of the topics:

May 7, Practical Software Quality and Testing, Las Vegas
May 13, Web Services Security and SOA Conference, Baltimore
June 24, SOA World, New York

I hope to see you at one or more of these events.

Testing topic

Next Wednesday at the Practical Software Quality and Testing conference I'll be giving the opening keynote on "Meeting New Challenges in Testing Service Oriented Architectures." We have been doing a lot of work with our customers recently on SOA testing strategies, and in the course of that work have built some technologies to help, in particular interface simulation tooling.

The challenge in moving to an SOA environment is to find a good way to validate service contracts. A good service contract is key to a successful SOA, and consequently the focus of SOA governance (and I mean this in the logical sense, as in the desired result of a successful governance effort).

One of the great benefits of a good service contract agreement is that the work to implement the contract can be divided between teams developing the application requesting the service and the team developing the service being requested (especially when such a service gets reused). But then it's necessary to ensure that all of the teams involved in such a distributed or divided development effort interpret the service contract the same way. This is the goal of SOA testing, and the summary of the additional testing challenge SOA adds to an IT environment. That is, getting the all-important service contract from definition to successful deployment.

As with any testing strategy, the sooner errors can be caught, the easier it is to fix them. Because an SOA environment introduces new artifacts, and requires new techniques for developing services (especially reusable services), it also introduces new requirements for testing systems.

Security topic

Tuesday, May 13 I'l be giving the opening keynote at the Web Services and SOA Security Conference on "Handling Multiple Credentials in a Heterogeneous SOA Environment," based on this article that I wrote with Fred Dushin.

An interesting aspect of an SOA environment is that it can abstract away many of the differences among hetrogeneous IT environments using a common interfacing technology (e.g. IDL or WSDL). Lots of companies have Java, Microsoft, mainframes, and other types of systems that need to be brought together in new applications.

Such heterogeneity represents a challenge for security technologies, since each environment tpically has a different approach to security. And in some cases more than one approach. When a service request message touches multiple technologies, it usually means encountering multiple security domains, which have to be federated and mapped in order to implement effective solutions for single sign on and authorization.

A valuable tool for dealing with this situation is a data structure associated with a service request that can be used to pass along multiple credentials, in whatever format they happen to appear, endorsing them if they arrive from a secure source (such as an encrypted communication channel). This way the service provider not only has all the security related information needed to call out to a security server, the application can also tell which of the credentials arrived from a trusted source. This can help resolve questions about the relative significance of a credential when multiple are in play.

Another interesting question in the world of security is whether is should be it possible for a policy unaware requester to interact with a policy-aware provider? I mean, if the requester does not specify any security policy, but the provider does, should the services still be allowed to interoperate?

Middleware topic

At SOA World on June 24th (Tuesday, 2:30 pm) I'll be talking about IONA's view of the middleware world. We have recently packaged an interoperability solution which some of us are calling "middleware for middleware" and that others will call our "universal adapter."

The basic concept is that people have enough middleware already, and they don't need more middleware if all they need is to get their existing middleware based applications and systems to work together. Instead, they need just the right amount of software to service enable the existing middleware, reusing that middleware's communications protocol, data format, and qualities of service as much as possible. The IONA solution is configurable for small footprint and high performance, and supports multiple deployment options (in the same address space as the existing application, in a different address space at the requester side, provider side, or in the middle) for C++ or Java (and COBOL and PL/I on the mainframe). And best of all it is priced accordingly - you just pay for the plug ins you need.

I delivered a version of this for a Webcast last year, the link to which you can find on our Webcast page. You can also check out the presentation.

A configurable, micro-kernel based solution implementing the call-chain interceptor pattern just seems like the best approach to SOA infrastructure. It offers the best match to the widest variety of requirements and does not impose on the solution any of the architectural constraints you get with a hub-and-spoke, server based, or mid-tier solution.

And yes, it supports SOA testing and security federation...

The Artix Connect for WCF Beta Experience

Thu, 03 Apr 2008 18:15:45 -0500

A couple of days ago we announced the Artix Connect for WCF product, and posted a beta on our Website. Today I finally got around to downloading it and trying it out with VS 2005. I am very pleased to say that it worked the first time! ;-)

The kit comes with a sample project that uses two connections: one to a CORBA based application and another to a JMS based application. The CORBA software comes in the kit, and you can use just about any JMS -- FUSE Message Broker ( which is a supported version of Apache ActiveMQ) is the default, which is open source and freely available. You can run everything on the same machine, per the instructions, but to use Connect in a multi-machine environment you would just reconfigure the network addresses of the CORBA and JMS software systems.

The way I usually talk about this is that WCF is for connecting to all things Windows, and Artix is for connecting to everything else. More precisely, Artix Connect for WCF is a Java-* interoperability tool that can be used from line of business adapters in Visual Studio 2005 and BizTalk Server 2006. One of the things you can connect to is Artix ESB, which connects to Java and native Tibco, Tuxedo, WebSphere MQ, C++ applications, etc. You can also connect to Artix Mainframe for accessing IMS and CICS based applications. And finally, Artix ESB can also be used to Web service enable all these existing systems and more, so if you are a WCF developer you have a lot of options for connecting to virtually anything non-Windows, while still coding as if you were using WCF.

The user's guide takes you step by step through how to set up the CORBA and JMS servers, configure the line of business adapter in Visual Studio, uncomment a few lines of C# code, and build and run the project. And there you go. WCF talking to CORBA and JMS. It's pretty fast, too, once it's all up and running.

This is pretty exciting. I've been briefing reporters about it. I have a lot of friends at Microsoft (including on the WCF team), and have been blogging and talking about the recent interoperabilty announcements from Microsoft. Some folks have taken a "glass half empty" view but I am definitely in the "half full" camp. I think these are very positive changes in direction for Microsoft, and I am very hopeful that Artix Connect will be very positively embraced by the Microsoft community.

Anyway, if you get a chance to try it out, let us know what you think. We have a month or two before GA, so there's still time to change things (and yes, I know, the EJB connector still needs to be finished). I had a good experience with it, but I am very curious to know what others think.

Annoyances (2): CTO is not CIO

Mon, 31 Mar 2008 16:25:13 -0500

Would someone please tell everyone who has apparently acquired my name and phone number from a marketing database service that I am not IONA's CIO?

I drive the company's technology strategy and our standards participation.

I do not buy servers, database management systems, operating systems (well, except for myself), phone systems, backup systems, SAP consulting, or broadband services. You can take me off your spam list.

My role is advisory -a classic software industry CTO position. (I do realize that a lot of CIOs are using the CTO title now, but that doesn't mean we have to forget the classic definition, does it?)

I do not hire developers (especially not in India!) and I do not hire consultants or market research companies. I do not buy mailing lists, I do not need help optimizing my Web site, and I do not want to attend your executive event. You can stop calling.

I am very sorry about all of that, since I know what it's like to cold call. But unfortunately, I can't respond to any of your numerous voicemails and emails, because that would just make things worse. And I'm sorry but I can't refer you to someone else, either.

Maybe you can get a refund from whoever sold you my name. If someone has included my name on a list of sales prospects, that is an error -perhaps it's simply list padding.

I also wish I had a good way to let you know - unfortunately I also doubt very much that you're reading my blog.

Microsoft Interoperability and Open Source

Thu, 20 Mar 2008 11:47:17 -0500

At EclipseCon this week Microsoft announced cooperation with Eclipse, among other things supporting the SWT technologies on Vista's presentation framework, effectively allowing Eclipse developers to generate GUIs for Vista.

Like many of their recent announcements concerning interoperability and open source, some observers are enthusiastic while others criticize the fact they didn't go farther and suggest they never will. However to me this continues to be a glass-half-full situation, in which take Microsoft's efforts in the context of their culture and history. These are big steps for them, and I think they represent a serious and significant change.

Last fall I attended an ISV event at Redmond, which we were invited to because of our interoperability solutions bewteen Microsoft, Java, and other environments. I couldn't help but notice that Ray Ozzie's name was mentioned several times by the presenters. That's why I made the 2008 prediction for Sys-Con that I did about Microsoft and the enterprise. It seemed to me as if Ray Ozzie's influence is starting to be felt. Burton Group analyst Peter O'Kelly was also quoted as saying so in the InfoWorld follow up article.

Does this mean that Microsoft is starting to become more serious about their interests in interoperability and open source? In the past I always got the impression that it was hampered by the fact that it would imply their recognizing the legitimacy of a platform other than Windows. Perhaps Ray Ozzie is able to bring a helpful external perspective. Perhaps reality is sinking in that the world of heterogeneous platforms is unlikely to change.

The main news for those of us offering interoperabilty solutions is that Microsoft is opening up some of its internal APIs and publishing their proprietary extensions to standards, which will make it easier to integrate with their products.

They are also allowing “reasonable licensing” terms on their patents – not sure how much benefit this is but they have also loosened up the terms and conditions under which other vendors can develop products that “infringe” on a Microsoft patent – i.e. they don’t have to get a license up front now, but instead only have to negotiate a “reasonable” fee when they ship.

The recent steps toward improved interoperability support and improved relationship with open source communities may strike some as insufficient or incomplete, but to me they represent a signficant change in tone and strategy for Microsoft.

Epitome of the Style

Wed, 05 Mar 2008 15:50:43 -0500

Yes, they are both about 60 years old, and many in the audience were, like me and my brother Tor and our friends Merrill and Brett, no longer the young rock 'n' roll fans we once were.

But this nonetheless had to be the epitome of classic rock, Steve Winwood and Eric Clapton together, playing everything from the blues, to soul music, Traffic, Hendrix, Blind Faith, Derek & the Dominoes, and various solo numbers from each other's back catalogues. I have heard and seen many comments to the effect of this being the best concert ever... Certainly one of the best concerts I've ever been to. The two masters are each great in their own right but clearly enjoyed playing together and just nailed tune after tune after tune...with a great rhythm section, no big band or backup singers this time. Like one of the reviewers said, this was a different Eric Clapton than with Cream in 2005. Much better - and it has to be Winwood, and the result of a collaboration rather than competition.

I know I am late, at least in blogosphere time, in posting this but better late than never. Same with putting the videos up on You Tube.

Actually quite a lot has been posted on You Tube from the three nights, and many comments as well. Here's the most popular one that I posted:

Excerpt of Winwood & Clapton playing Hendrix tune "Little Wing" at MSG 2/28

This was definitely one of the highlights. "Voodoo Chile" and "Double Trouble" were two others - with Clapton absolutely nailing two of the best guitar solos of all time during the latter. The organ/guitar jam on the former was incredible, and made me wish there had been more organ solos. Winwood is so versatile though - such a great singer, piano player, and such a great guitar player, trading solos with Clapton on several songs, including "Cocaine" and "Dear Mr. Fantasy" (the encore last Thursday) and certainly holding his own. In fact not many guitar players are better than Winwood, Clapton being among a relatively small handful.

Our seats were great, about halfway up the risers toward the back of the MSG floor, on the right, with a clear view of the stage. One piece of bad luck - a very loud mouthed New Yorker and his friends sat directly in back of us. "OH MY GAWD!" "HE STILL HAS HIS SEVENTIES VOICE!" "HE IS JUST AS GOOD AS WHEN WE SAW HIM IN THE 70s MAYBE BETTER" "WOW IT'S 'LITTLE WING' " etc. and on and on at the top of his lungs, as if he could onl enjoy the show by constantly talking over it. No doubt he will be bragging to all of his friends what a great show it was, despite the fact he didn't really listen to it.

I can't wait for the DVD. I hope they get all the songs on it, not like Crossroads where they cut out a bunch of stuff, and changed the order on some others (not that anyone who wasn't there would notice, but still).

Afterwards we walked up to Times Square to an excellent Belgian beer bar to rehash the evening.

The weather was a bit cold and windy outside, but the memories are still warm.

Is Stonebreaker Right? Time to Reinvent the Database?

Wed, 13 Feb 2008 05:40:08 -0500

At HPTS this past October, Michael Stonebreaker delivered a presentation called It's Time for a Complete Rewrite.

The main point seems to be that the general purpose relational database management system has outlived its usefulness after 30-40 years, and the market needs specialized database management systems to meet current application requirements.

A lot of these topics were covered in an interview published in the Sept/Oct issue of ACM Queue. However, Mike stops short of describing some of his new proposals for these specialist databases. Last Monday a lot of this was discussed at the New England Database Day session at MIT, where Michael now teaches.

It looked to me as if about 100 people showed up, and I believe they said a majority were from industry. The presentations were very interesting. A good summary can be found here.

A highlight was certainly Dave DeWitt's presentation on Clustera. Despite the fact I've been taking an interest in what Google and others are doing around the new "scale out" architectures, I had missed Dave's blog on why Map Reduce isn't so great. He included some of the points in his presentation, but to me it was more of a defense of the current RDBMS than a realistic criticism of Map Reduce in its own context. I am oversimplifying, I'm sure, but a lot of it sounded like "you can do this with a relational database if you wanted to, so why are you bothering with something new?"

Personally I think this kind of misses the main point, which is to consider what advantages can be gained by doing more in memory and disconnecting the whole persistence thing. Another way to put it is that the industry has been focused for years on the fastest way to persist data in order to ensure the best reliability and consistency possible, and do as much automatically as possible and avoid human intervention - the kind of stuff I had to do before transactions were widely available in database systems, i.e. go in by hand and find any partial results that occured due to machine failure and back them out.

But if we were to break that assumption, and say that manual intervention might be ok in some cases, and everything does not have to be done automatically, we could gain some advantages in performance and overall capabilities to handle large amounts of data more effeciently.

I definitely agree it's time for some new thinking in the database and transaction processing world. The specialized database ideas have a lot of merit - column-oriented databases to improve data warehousing and decision support, streaming databases for event processing, embedded databases for higher performance and better flexibility, and in memory databases for improved update latency - but the most interesting point for me is changing some of the key underlying assumptions.

For example, using memory based systems and thinking about persistence as a second level operation, disconnected or asynchronous from the memory update. Assuming failure and designing systems that fail often to take advantage of lower priced hardware. After all, even the biggest and most expensive systems fail - including software, of course. So why not assume it? And the idea of a central controlling authority, such as a central database system of record, root transaction manager, common system adminstration domain - the Web based designs are clearly showing the need to redesign and rearchitect not only our system software such as databases, TP monitors, and application servers - but also our enterprise applications.

Have We Got it all Backwards with Software Assembly?

Fri, 01 Feb 2008 14:00:00 -0500

I am as guilty of this as anyone else. Back in the 90s I was on a big project to standardize enterprise software. We wrote a few papers about it, and a chapter in a book. We often used the "Henry Ford" analogy, which relates to the impact standards for interchangable parts had on hard goods manufacturing.

The Henry Ford analogy says that the hard job in mass assembly is getting the interchangeable parts standardized - thereafter creatng the moving assembly line is the easy job. Ford pulled it off with the significant market success of the Model-T and changed the world.

In the original story (which the link directly above summarizes), the crucial quote for us was:"The key to mass production wasn't the continuously moving assembly line, as many people believe, but rather the complete and consistent interchangeability of parts and the simplicity of attaching them to each other."

But of course in the updated book, Toyota further changed the world from craft to mass production (i.e. Ford's achievement) to lean production. In software however we are still struggling to achieve mass production, never mind lean production.

The application of the Ford analogy to software is that if you can standardize application programming APIs and communictions protocols, you can meet requirements for application portability and interoperability. If all applications could work on any operating system, and easily share data with all other applications, IT costs (which are still mainly labor) would be significantly reduced and mass production achievable.

The industry has seen many efforts in this direction fail, or only partially succeed. Today's environment is better than the early 90s, but we still have incompatibilities across various editions of Java, enough differences among J2EE application server implementations to inhibit easy migration among them, and of course a signficant level of incompatibility between enterprise Java, Microsoft .NET, and IBM mainframe environments. Applications that want to leverage the best of breed across these enironments typically have to do a lot of craft, i.e. hand coding.

Seven years ago I remember thinking Web services and XML might finally solve the problem, but perhaps because of the way the specifications were implemented (basically adapting to existing technologies) in the end only a partial soluton was achieved. Yes, interoperability is improved compared to what it had been, but it still requires too much hand coding.

Even though I've been working towards the "Henry Ford" analogy for more than a decade, recent exposure to inversion of control concepts (e.g. Spring and Guice) and OSGi makes me think the mass production analogy is backwards for software after all.

The Ford analogy has played out in software typically by positioning the assembly problem as the easy part of the job and creating resuable services for assembly as the hard part of the job. I can't tell you how many times I've heard business process modeling and orchestration tools pitched at "business analysts" only to discover the proper use of the tool requires someone who can actually code.

The easy part should be developing the reusable services. The hard part should be their composition and assembly.

Corporations around the world are squeezing IT budgets, which means looking to reduce labor costs. Many are turning to outsourcing to China and India, and others are looking to hire college graduates in place of highly paid (and more highly skilled) coders.

But almost by definition the Ford analogy can't work. You cannot really get lower skilled, untrained developers to tackle sophisticated problems such as component reuse. They can create simple objects incorporating business logic, and to use one description, the plainer the old Java object (POJO) the better.

What we need are not simple tools for business analysts to compose services into flows. We need sophisticated tools for architects and designers to import POJOs and plain old anything else, check them for conformance to the plan, and fix them up for deployment. What's the right analogy here? Farming?

OSGi Enterprise Update - Florida meeting

Tue, 15 Jan 2008 16:25:12 -0500

I came back last Thursday evening from the OSGi Enterprise Expert Group meeting hosted by IBM in Boca Raton for the IONA (internal) services summit, where I had the privilege of assisting in the award to Adrian Trenaman of his promotion to Distinguished Consultant. It was quite a moment for the IONA services team and for Ade as well I think.

Yes, we really met in the Ozzie Osborne Conference Room - but it's not named after "Ozzy"

And then on to the annual sales kickoff, where I gave a presentation about big picture industry trends (largely based on what I learned at HPTS), where we fit, and how we're going to incorporate innovation into our product planning process. Fittingly, Ade is a part of the new technical vision team we started last August for that purpose. So he was a significant part of both events. Well done, Ade! (see also his interesting discussion last year with Ted Neward at the Server Side on contract-first)

Anyway getting back to the meeting, a lot of the discussion concerned the metadata design for the distributed OSGi proposal. Among the major requirements for enterprise OSGi is distribution - meaning the ability of an OSGi service running in one JVM to invoke a service running in a JVM in another address space, as well as the ability for an OSGi service to invoke a service running in another distributed computing environment and vice versa (i.e. accept a service invocation from another distributed computing environment).

We had agreed at an earlier meeting to try to reuse as much metadata from SCA as possible, given that SCA had already solved many of the same problems. However there are also significant differences between SCA and OSGi, especially in terms of the development environment and the differing assumptions around dynamic vs. static wiring. It should be possible in OSGi for example to have a client invoke a service that is dynamically provisioned. And finally, the design center of SCA is assembling services and attaching policies to them in a way that should be language and runtime independent. OSGi's design center is around solving specific deployment, modularity, and dynamic lifecycle issues with binary Java files.

One of the larger context debates in this area was called "XML vs properties" and I think we came up with a great kind of middle ground in which you can use properties for simple distributed computing configurations and "call out" via URL to a more complex XML file when necessary. We are still working on the proposal but we are planning to start a reference implementation soon using ServiceMix, and through that process refine the proposal further.

During the meeting we also got a great demo from Adrian Colyer of dynamic Spring (aka Spring-OSGi). His spec is really well organized, clean, and reads really well. I am pretty envious of that, although as Hal Hildebrand pointed out, the Spring-OSGi work is by now much better known territory, and the document is more a reflection of reality than an unproven design (it is still early days with distributed OSGi, that's for sure).

We also spent a good bit of time discussing the proposal for an optional discovery service that should allow any existing discovery mechanism to be included into the distributed environment. For that matter, it's worth emphasizing (since this seems to be fairly consistently misinterpreted or misunderstood) that we are not proposing to invent any new distributed computing mechanism. The goal is rather to define how any distributed computing mechanism could be configured and deployed into an OSGi platform - and hopefully even define some metadata sufficient for using more than one successfully in combination (i.e. interoperability among supported distributed computing mechanisms).

Neither is the goal to choose or favor any distributed computing mechanism over another - we should be able to support on a fairly (if not totally) equal basis mechanisms such as RMI, RMI/IIOP, SOAP, JINI, SCA, JBI, J2EE, JAX-WS, CORBA, JMS, etc.

Another important part of the EEG activity is coordination with Eclipse and the Equinox project, which is an implementation of OSGi SP R4. Other open source and commercial implementations of R4 exist (see Apache Felix and this list of certified products), and representatives of most of them participate in the EEG, but Eclipse has recently started moving toward the runtime space.

After the Runtime Summit Jeff mentions in his blog, I sat and talked with Cote of Red Monk about what's going on. The OSGi folks saw this and told me that the statement about "OSGi being the only other organization legally allowed to create Java specifications" is a bit broad. (Perhaps someone just told me something along those lines when trying to recruit IONA to join ;-).

Anyway now that I'm on the Board I've been able to get a look at the actual paperwork - the agreement between Sun and OSGi that allowed the OSGi Alliance to be formed to progress JSR 8 outside of the JCP. I can't quote from it, but what I can say is that it is somewhat ambiguous. Saying what I did is probably a bit of a stretch. It is more accurate to say the OSGi Alliance is allowed to progress that original JSR and related work.

For the record, IBM has built a new facility in Boca Raton, which is where the meeting actually was held:

View Larger Map

And here's a photo taken right outside the entrance, which is around to the rear of the building.

Ahhh, Florida. :-) I was scheduled to go back there yesterday after the kickoff presentation (which was Sunday morning), but the snow storm caused Florida flight cancellations (Jet Blue, I still love you) and I ended up driving home through the worst of it. Yes! I can still drive in the snow! I ended up joining the meeting by phone, but I missed getting back down to those balmy breezes, palm trees, and fluffy white clouds.

The drive home yesterday did not look like this.

Artix and Fuse updates

Wed, 12 Dec 2007 19:46:19 -0500

Update 12/14/2007 - A great article from Rich Seeley on the hybrid approach initiated with this weeks' releases.

My favorite part of this week's announcements around Artix and FUSE is the support for enterprise integration patterns (EIP).

When I was in Tokyo a couple of weeks ago I heard that a former colleague, now working for Accenture, has recently been giving presentations about the applicability of these patterns to addressing various enterprise IT problems. It actually makes a lot of sense that people would want to have their software products directly support the development and deployment of common patterns - whether in the integration space or not.

EIP support in both Artix and FUSE is derived from the Apache Camel project. As we say around IONA, and as I hope everyone knows, a camel is superior to a mule... ;-)

Illustration of some of the Integration Patterns Now in Artix

I also think it's great that Camel is using the domain specific language (DSL) approach, since I've been a fan of DSL for a long time, although two years ago I was characterizing DSLs in opposition to UML/MDA. Since then I believe MDA has turned more toward modeling than executable code, which is good, and annotations and aspects have kind of arisen to take their place.

Eclipse tooling shapshot for EIP

At the Eclipse Board meeting here in San Francisco I just presented an update on the SOA Tools Platform Project, including the snapshot accessible via the above link that illustrates what we're working on in that project to generically support EIP through open source tooling.

I may be wrong here but I think the convergence of these two trends is going to be huge - the identification, characterization, and codification of EIP - and the specialization of DSLs to solve specific computing problems.

SOA in China

Tue, 04 Dec 2007 17:45:39 -0500

In late October I visited China to discuss SOA with several telecom customers. I am a little behind on my blogging, so bear with me. Like they say, sometimes it's hard blogging about the things you do when you are so busy doing the things you should be blogging about... ;-)

Saturday morning, just before flying home, I had a chance to visit the Forbidden City. Lulu from IONA's Beijing office was kind enough to accompany me so I wouldn't get lost. I really recommend it if you are in Beijing, it's a great old palace with lots of temples and great museums of clocks, manuscripts, bronzes, and other things.

Me in front of a ceremonial hill at the back of the Forbidden City

It's amazing how quickly Beijing changes. I was last there about a year and a half ago, and since that time three subway lines have opened, and an entire new wing was added to the hotel where I stay. The big question of course is whether the city will be ready for the 2008 Olympics. I think it will be a close call, there will be some problems, but they will pull it off all right.

And they are apparently keeping up with the latest IT trends. Anyway SOA and Web services are pretty hot in the telecom area right now, especially within the new Service Delivery Platforms (SDP). Just as they are in other parts of the world, in China service orientation and Web services are seen as techniques and tools for improving the effeciency of delivering new telecommunication services to market.

Sometimes you hear things like "the Chinese market lags behind Europe and the U.S." but I'm not sure that's true. For one thing, that's a very broad generalization that implies European and U.S. companies are all at the same stage of advanced thinking, or that no Chinese companies are thinking about advanced topics.

I spoke at an SDP conference this past June in Budapest - I was asked to give an introduction to Web services for the telecom industry. And I was able to use most of the same material for the customer presentations in China.

It may sound a little strange to still be giving an introduction since Web services have been around for about 7 or 8 years. But it has really been the business systems folks -- the back office systems for billing, inventory management, order management -- who have been investigating SOA and Web services. The network management folks (the ones who deliver the calls, troubleshoot the network, and manage the new services) are just now getting started.

WSDL's unique ability to abstract multiple protocols (IIOP, HTTP, JMS, etc.) and data formats (XML, CDR, ASN.1/BER, fixed format etc.) and categories of software systems (i.e. message queuing, application servers, database management systems, and packaged appications) is a big part of the solution since CORBA is used in most existing telecom network management products, and lots of telco specific formats and protocols are in use.

Maybe because they've been using a lot of CORBA, including notification for event handling for switch failure and other types of alerts, I got a lot of questions about performance and the impact of the additional overhead of XML processing. In fact I got this so many times I started to think that another vendor was going around saying that their Web services products didn't perform very well! In our case we certainly do also see the usual additional overhead involved in sending lots of XML text, but if someone needs it to go faster we just change the configuration to run SOAP over IIOP instead.

As promising as it sounds for telecom network management applications, no one really knows exactly what the definition of an SDP is. TMForum is working on this in its SDF initiative. But there's also activity in IEEE around NGSN and quite a few other similar initiatives.

What I heard at the conference in Budapest is that the telco carriers are worried about competition from Skype, Google, Yahoo, and other companies delivering innovative new products and services over the Internet. The carriers still have "carrier grade" networks, and if they can open them up using services, perhaps they will be able to attrack some of that innovation. At least that's what I heard one of them say.

But of course the larger issue is developing and delivering new telecom services such as Voice over IP and TV over IP, TV on cellphones, expanded multimedia delivery to the home, etc. And here is where SOA and Web services can play a key role - especially if we can get them to go fast enough (and I believe we can ;-).

Christmastime again

Tue, 27 Nov 2007 21:42:15 -0500

Update 12/6 - Number 1 on Google images today! "One little Christmas tree" really did light up the world, at least the world wide Web!

Here in Tokyo the Christmas decorations are up, as they were back home near Boston. It is certainly something shopkeepers seem to agree on around the world, Christmastime is good for business.

A couple of weeks ago I recieved the second formal request for publishing rights to a photo of the family Christmas tree I posted a couple of years ago, along with a list of "why can't we all get along" type wishes in the spirit of the holidays.

The first was from someone publishing a safety brochure. Apparently they wanted to warn everyone not to do what we did and put too many lights on. But I'm really not sure - they did not really give a good answer. But I figured what the heck.

The second was from a family Christian organization. Apparently they wanted to show what a nice family Christmas tree might look like. They both said they'd credit me and cite IONA as the copyright owner. I actually have no idea who owns the copyright in this case. I would have thought it was me, but perhaps by uploading it to the IONA sponsored blog site I implicitly turned over the rights. As I said, I have no idea.

But this time, I started wondering what was going on. I suspected a subtle scam, maybe some new kind of phishing. As anyone with a blog knows, blog spam is out of control, and maybe this was some new way to get some links to a nefarious Website. But I really didn't see how it could be. Finally I thought of checking Google.

Sure enough, there was the old photo of the family Christmas tree, popping up at number 4 (at least today) on an image search for 'Christmas tree.' After a while it struck me that my photo was the only one on the page that didn't look staged, or professionally done. What I still don't know though is how many times the image has been used. If I got two formal requests, how many just downloaded the photo and used it, without bothering or thinking to ask?

ps Happy Christmas

pps Can we keep working on those items? Or maybe we need a new list? For example, now we have OSGi EEG and Java EE 6 to reconcile for 2008

The Second Wave

Fri, 16 Nov 2007 16:42:23 -0500

The IT industry is in the middle of a signficant transition. At least one article has already labeled it IT 2.0. I give Dr. Elizabeth Joyce (a former IONAian BTW if I'm not mistaken) some credit for noticing the change in IT, but the industry grokked the change in emphasis from data to information about 20 years ago. That's why it is now called "Information Technology" instead of "Data Processing."

It is also true, as she points out, that IT is becoming more business focused. But this is a symptom of change, not the cause. The cause of all the changes is the result of the first wave of adoption drawing to a close and the second wave starting up.

The first wave was about adopting computers in business - automating previously manual operations. This activity was based on a strong ROI that justified a heavy initial investment in hardware, software, and application development.

The second wave is about improving upon what was created during the initial adoption phase, and is based on a very different ROI. Automating a previously manual business function is a very different activity than improving upon that automation.

Let's look at some of the clear signs of change in the industry:

SOA adoption
Vendor consolidation
Open source commoditization
Offshoring and outsourcing

SOA Adoption

Depending on who you ask, SOA has been around for 10-15 years. For sure IONA has customers who have been actively developing SOA-based applications for 4, 5, and even 8 years. So we know it has been around for a while, but it has been on the peripherphy until now. Why now?

Gartner says the term first appeared in a 1996 research paper by Yefim Natis. I also know that the Credit Suisse SOA has been in production since 1998. So it's clear SOA has been around for a while, but why is it so popular now, as opposed to 10 or more years ago when it was introduced? Don't IT trends typically follow the reverse pattern?

I believe the answer is that the industry has finally reached the point of maturity at which it makes sense to adopt it. Instead of looking forward to the next function to automate, enterprises are looking back at the heterogeneous spaghetti-like mess resulting from the heads-down rush to computerize. It's only now that the end is in sight that we can start to take a look back and assess what was done - and try to find a way to improve it. I.e. adopt SOA and reuse it.

Consolidation

Larry Ellison gave his take on this to Business Week in a 2003 interview. I think he was right about some things, but obviously wrong that software innovation would primarily come from large companies and that the startup wave is over.

But it interesting that other industries have also gone through an initial adoption wave and then gone through a consolidation stage. Making it a clear sign the first wave is drawing to a close. No doubt many of the transition changes are painful in starting the second wave. But unlike Larry Ellison, I believe the established enterprise software companies are not immune from that pain. In fact they may bear the brunt of it in the end. They are not changing their approach - they are stuck promoting their old products, and in their old business models. Consolidation is at least in part a defensive move. Industry disruption does not come from the establshed players.

Commoditization

Another clear sign of change is in the commoditization happening in open source software. We don't need more new features and functions in our operating systems, database management systems, programming languages, and middleware. We need cheaper and better implementations of them. This is a clear sign we have reached a critical mass of core features and functions, and the industry does not need a major new feature or function in enterprise software any more than an office worker needs a new feature in Excel or PowerPoint.

To continue the office automation analogy, at one time new features were defninitely needed in Word so that companies could justify its use in place of special purpose word processing machines, but no longer. That war is over.

Similarly, the open source phenomenon is all about commoditizing well known and well established software features and functions, and increasing its overall value. During the initial adoption phase with a strong ROI over replacing manual processes, it made sense for vendors to compete based on their IP - someone was, after all, going to come up with a better way to implement a transaction. Now that we are done with the first wave we can focus on how to produce software that accomplishes the same things more cheaply and effeciently.

Outsourcing

Here the correlation between the first and second waves is pretty clear. Durig the initial adoption phase the cost of labor, still the largest cost of IT by far, was not a signficant issue since we were replacing manual processes with computers. As we start on the second wave, reducing labor costs will continue to be important. Companies are struggling to reconcile the costs of IT with their business strategies - it's very hard for many companies to understand exactly how their IT spend influences the bottom line.

A better way of doing IT, such as a division of labor through the adoption of service interfaces (for example) and the trend toward configuration based development (e.g. Spring and OSGi) are clear responses to the need to do more with less - to increase the proportion of the contribution from lower cost IT labor, in other words.

Summary

Remember when all we cared about was finishing one application so we could move on to the next? Should we automate order entry next? Inventory management? Manufacturing planning? How about removing those old office automation machines and replacing them with PCs? We are so done with all that.

Now it's all about how to improve what we already have, since what we already have is pretty much enough. I mean we have automated pretty much all of the previously manual operations that can be automated. Ok, there are still a few left, but what's left is of marginal impact.

The major point is the difference between a foundational investment and an incremental investment - the difference between the first and second waves.

A foundational investment establishes the industry - in this case creates the initial applications - based on the initial wave of adoption of computers. The second wave of adoption, in which the balance of the investment shifts to improvement, is well underway, with clear evidence in the major changes now taking place.

Consequently new and different approaches to IT are emerging - such as SOA - to focus on improving what's already there. And a new kind of enterprise software is also emerging, one that is better aligned with the changes of the second wave - a lightweight, configurable container - that gives the right set of features and functions for the right price point.

Oracle/BEA - Industry change highlights need for better software

Mon, 12 Nov 2007 19:04:52 -0500

Consolidation is a sign of change.

The proposed BEA/Oracle merger -- which by the way seems inevitable -- is a clear sign that the IT industry is in the middle of significant change.

In the beginning it was all about creating applications, putting in place the automated systems needed to run businesses more efficiently than the manual processes and procedures that predated them.

Now that this is done, or nearly so, it's time to improve the effeciency of those existing applications - some of which have been in place for 30 or 40 years. It's not about new applications, or databases, or application servers for that matter. It's about finding a better approach -- SOA for example -- and about designing and building better enterprise software to help get the most out of what's already there.

Every industry goes through a maturity cycle. With software we are right in the middle of a big inflection point. The original business model - the one that produced 80% margins after you covered your cost - is rapidly fading as open source comes to the forefront. The bigger companies are hit hard by this, and start to merge and consolidate to increase sales volumes and streamline production.

But that's not enough. What many of these big guys miss is the fact that the software itself also needs to change - as the industry changes the product designs have to change to keep pace. Nobody needs more of those big stacks of enterprise software; no one needs more of those big server hubs. This is of course very hard for the big guys - they are stuck in the cycle of responding to their current customers' requirements for improvements and enhancements to their existing products. They are caught in a classic "innovator's dilemma." And instead of innovating, or facing the pain of major change, they keep busy merging and consolidating.

It should be clear by now that customers have enough of the old expensive middleware. What they need now are lightweight, less expensive products that will help them extend applications into the modern world (e.g. RIA) at an appropriate price point. It should not cost as much to improve an application as it did to create it in the first place. A different approach to software is needed.

Following the initial automation push enterprises are left with stovepiped applications, heterogeneous environments, and outmoded approaches. In other words, we need tools and techniques to make what's already there work better. And that is not what the Oracle/BEA merger is about.

We need software to unlock innovation and improve agility, not the same old product designs that were good 30 years ago just because someone's stock price depends upon it. We need software suited to today's challenges - a lightweight, configurable microkernal based approach to distributed SOA infrastructure. In both open source and commercial software flavors ;-)

HPTS 2007 - time to reinvent everything?

Sun, 14 Oct 2007 14:43:43 -0500

It's been a busy week between HPTS and the most recent OSGi enterprise expert group meeting. This entry is about HPTS. I'll post another one about the OSGi meeting.

It was pretty clear from the presentations and discussions at last week's HPTS Workshop that database and middleware software vendors are failing to meet the requirements of Web businesses, both large and small.

My room in the Julia Morgan designed Scripps building at Asilomar Conference Center

During the tribute to Jim Gray Tuesday evening Pat Helland showed a great video interview with Jim in which he mentions the need to scale things out, not up.

Pat Helland introduces the tribute to Jim Gray

The "scale up" thinking has a limit - there is always such a thing as the biggest box you can buy. But "scale out" does not have any limits - you can always add more computers to a cluster. The presentation from Google in particular was a really interesting explanation for some of the techniques they are using to achieve a massive scale out. Pat's summary of it is something like "people are willing to trade consistency for latency." And because current products are designed around the "scale up" concept, they don't tend to work very well in the modern "scale out" environments.

One disappointment was that the scheduled presentations from Amazon.com were pulled at the last minute. But the other presentations from Yahoo, Betfair, PayPal, Second Life, Salesforce.com, Rightnow, Workday, Google, and eBay all made it pretty clear that Web based businesses are developing custom solutions because off the shelf software currently does not cut it for them.

Aaron Brashears of Second Life

Mike Stonebreaker's presentation contained perhaps the most direct challenge to the status quo, arguing that industry changes during the past 30 years have invalidated the relational database and SQL. This provoked some lively discussion since among the attendees were several people who helped define and create the relational database and build DB2 and Oracle. One of his arguments is that the relational database just isn't fast enough for modern requirements.

Some great discussions broke out between the representatives of the Web companies and the folks working for established software companies. For example, one of the Web guys commented that they did not want to have to buy the whole set (i.e. complete product) when all they really needed was a paring knife, yet the only choice they are offered is to buy the whole set. (And here OSGi has some potential to change that dynamic - but more about that in the next post.)

"Bloatware" is definitely part of the issue. Ironically so are many of the things a lot of us have worked to define, create, and promote during the past two or three decades around guarantees of atomicity and isolation. None of the Web companies use distributed two-phase commit, for example. They only use it to deque an element and update a database in the same local transaction. So much for all that work on WS-TX! ;-)

Anyway, the idea seems much more to be how to replicate data across multiple computers, so that if one of them crashes another one can take its place, to write programs assuming that failures will occur early and often (since they are going to occur, why not assume they will occur often?), and to allow the replacement of disks and computers as needed without taking the application down.

It will be very interesting to observe over the next few years the extent to which the ideas and techniques in the custom built solutions become more widely adopted and incorporated into commercial products. One of the inevitable questions, as raised during the discussions, is how broad the market is for such things as Google's file system and big table, or Amazon's S3 and Dynamo.

In the old days we always talked about moving mainframe based systems to distributed environments, but maybe our mistake was in trying to replicate the features and functions of the mainframe. These new "scale out" architectures may finally accomplish it.

Some of the way too many photos ;-)

View from outside the dining hall
Me and Ed Lassettre
One of the boardwalks through the dunes between the conference center and tbe beach
View of Monterey Bay from one of the boardwalks
Down at the beach, view from the end of the dunes
Down at the ocean, the tide coming in around the seaweed
Up on the path someone walks his dog
Through the rocks, the clouds and the surf
The path beside the beach, looking north
Surf and mist, looking southward
The Julia Morgan designed chapel, where the workshop is held