Connectathon Day 4

I didn't keep a log of activities for today, since I was rather up to my ears. So, I'm going to take a few minutes to talk about the Connectathon in general.

One of the best things about connectathon in my experience is the collegial, problem solving atmosphere. We are here to make our products interoperate with each other. It's not about making ourselves look good, or the other guy look bad. The connectathon monitors don't care. If two parties in a transaction cannot make it work, neither one gets credit for it. The people working here truly do work together on solving their interoperability problems.

At connectathon we run into the same kinds of problems that we don't want to appear in the field. Here we have the people that would normally be providing Tier 3 support on such an issue present from all sides of the problem. The only danger here is in failing to pass a test, not dealing with irate customers or losing contracts. This environment is extremely condusive to problem solving, and most of the time before any customer would ever see an interoperability issue in the real world.

Another incentive is that many of us will be going on to demonstrate our work at HIMSS. This is an opportunity for many engineers to appear in one of the most high profile booths on the HIMSS showcase floor. Traffic at HIMSS through the showcase booth has exceeded 8% of attendees for the past four years, and has gone as high as 12%. If these numbers seem small, realize that an Anchor booth is doing good when it draws 3% of the attendees. The IHE showcase at HIMSS is sigificantly smaller than those booths, and has traffic all week long. It also recieves quite a bit of high profile traffic, many in key decision making posts and government policy positions.

I offer the following picture as proof that connectathon makes interoperability a priority over everything else (reproduced with permission of those present). These engineers were working late into the evening making sure their systems worked together.

Connectathon Day 3

Dinner last night was down the street from the hotel after a brief reception honoring the current Director of IHE for HIMSS. After dinner and a short cab ride back to my hotel, I spend several hours catching up on the "day job" and e-mails. I spent fifteen minutes trying to VPN to the office before figuring out that my network card was still configured for connectathon. That was at 2 am. As usual this week, I've stayed up too late.


So, now it is 9:04, and time to reset the network connection for the connectathon.


Just to give you an idea of the processess I have running on this laptop for connectathon:

• Microsoft SQL Server -- Hosting a database I have containing a lot of standards information.


• Oracle SQL Server -- Hosting my EHR application's data


• 2 different Copies of Eclipse -- One used for the web services I'm testing, the other used for interfacing.


• My favorite XML Editor -- which supports XSLT debugging and Schematron validation. I do a lot of both. I used to work with one of the the co-chairs of the XSL workgroup in the W3C, and in the office next to an author of XPath. It has an Eclipse plug-in, but I've found I run out of memory in Eclipse a lot sooner if I use that.


• Acrobat Standard -- Viewing the IHE profile and some JSTL documentation


• A web browser (with about 3 tabs open) -- Connected to NIST's CDA document validator, this blog, and the IHE PCC Web page.


• Microsoft Word, Outlook and Excel -- more documentation, my e-mail containing various connectathon communications, and a spreadsheet I'm using to track testing.

I upgraded the memory from 1Gb to 3Gb last year just so I could get through testing without waiting forever for processes to swap.


9:22 Hmm, I seem to missing some data. Ah, it's the Change proposal we applied this year to address sections that have nothing to report (e.g., no allergies). Need to go fix that.


9:45 "Does a section require a title?" gets fired at me from a connectathon monitor. "Yes, I believe so, let me look it up... wait is this a section defined in CCD?" "Yes." "All CCD sections require a title." However, I note that we need a Change proposal for the PCC Technical Framework to include that requirement for all sections it defines, because not all of them come from CCD.


9:46 "How many PIX consumer tests do we need to do?" "Three." "Ok, I've already done two, I'll go work with ...", "Nope, you can do that later, go work with ... to get them going."


9:50 Overheard: "I basically blew up vendor A's actor B because I didn't have an (indecipherable technobabble)"

10:15 I've usually done this weeks in advance, but this week I'm still writing code for my stretch goal. You aren't supposed to be doing that at connectathon (as one of my mentors observes). However, in my own defense, I was going to drop this profile. Someone asked me to stay in so they can get credit for it. I've been in similar situations before, so I'm going to do what I can.

12:12 Over lunch we discuss the structure of the Public Health Alerting scenarios for the demonstration. The CDC is obviously very interested in this scenario.

1:13 I argue HITSP terminology changes with one of my co-workers. I like simple words that engineers already understand. He thinks that other terms are better. We agree to disagree.

By mid-day, I have my stretch goal done (with one retest required). Since it's a CDA document, and I helped write the profile there's a lot of laughter that I got failed the first time through. Read the free manual...

2:02 This afternoon it's time to do T81 (InfoButton testing) for the public health alerting scenarios. I wrote the test spec, so now I have to follow it. But... but. There's some things I forgot about when I wrote it back in November. We'll deal with that tomorrow. For now, I just update the argument table and resend.

2:45 One of my test partners for this test just got back. He had to take a call (common), and also hadn't realized that connectathon is a casual dress event (unless you are a docent for the connectathon conference which was yesterday), so came back in his jeans. We test our scenario. After about ten code changes (most on my side, and a couple on his), we get it working. It took all of fifteen minutes. We both mispelled Notificiation the same way, but I had corrected it earlier when starting my T81 testing. On my side, it was a copy and paste error from the spec I wrote several months ago. I fix the spec too.

3:43 An engineer asks me what value should go in a particular message. It's in a profile I don't know, so I ask him what the profile says. "I don't know". Well, read the profile, then ask me... I follow up with him later to point him to resources that will know if he's not able to figure it out.

5:00 Time for the HIMSS demonstration meeting. I listen in from the other end of the hall. Technically I'm supposed to be there, but someone else is covering for me, and its the same sort of thing that I've heard countless times before. We also see again the unveiling of the dreaded scenario spreadsheet that will likely be updated several times before the end of the week. It's pretty solid by now so I need to look at it.

7:20 Still here, working through some other problems. One of the systems that we've been having problems with all week was misconfigured. The engineers worked it out and are on to the next issue...

Dinner tonight is at Brasserie Jo. They have a nice collection of Belgian beers, and some interesting food. I had the Shrimp Bag. It's shrimp and vegatables in a lobster sauce, wrapped in Philo dough and baked. Yum!

2:29 AM. Time for bed. Need to get up for an early Domain cochairs meeting.

Connectathon Day 2

One of several traditions at the connectathon is an evening of Sushi at Tsunami with a group of about ten-twelve people. This year we had to cab it over, but most years we pack ourselves into a large stretch limo for the trip there and back (The limo is cheaper than the three cabs it otherwise takes).


Afterwards, it's back to the room to catch up on the day's e-mail and current projects. This year my room has a wired rather than wireless internet connection. That's a bit of a bother -- I'll have to undo and redo my connectathon network configuration all week. I find myself thinking about how amazing it is what technological advances we take for granted. Five years ago, I might have had to check for internet access at all, now I simply assume it will be there and be wireless. It gives me hope that similar things can be done in healthcare.


Something I learned previously about Chicago, but which I forgot is that there are two Wacker Drives, one going East-West, and the other North-South. Unfortunately, there is a Hyatt at each one. I wound up at the wrong place this morning, but it was quickly fixed once I looked up from notes I was scribbling.

My morning winds up being filled with mostly hallway conversations. I get one person signed up to list out standards used in IHE profiles I don't know for some internal work. My manager is here today attending the connectathon conference, and we chat briefly.

At 9:20 I remember to reset by network configuration.

I have a chat with some test developers from NIST on XDS-MS content that was generated. It's missing some key pointers preventing it from being imported using one of the profile options. I track down the developers and give them some guidance on what they need to do to fix it.

I run into someone who wants to talk about SOA. We figure out what context the discussion comes from (HITSP, HL7, IHE or fill in the blank), and talk briefly about how IHE profiles are in many ways services in the SOA sense.

I connect up with a potential testing partner for one of my stretch goals and promise to create some content for one of the tests they would like to pass by tomorrow.

I go over the need for TLS in the ATNA profile with another small group, and explain that now is not the time to complain about how the profile works. If you've signed up for the test, you need to pass the profile as written.

I work with one of our teams to resolve a blocking problem. The solution seems to be nearly in hand, and we work with the product team to run a different set of tests while that problem is being resolved.

It's 11:20, and I have yet to run a single test of my own. Fortunately for me, these are not critical, but my testing partners would like to pass also...

At 11:46 I open 6 tests which need live monitoring. Got into a discussion of grouping between two profiles, so I pulled up the PCC wiki describing one of the profiles and its grouping with other IHE actors. After another 20 minutes of dicussion, we've just identified another place that needs clarification. Now it's time to track down a monitor for the tests I opened 15 minutes ago.

It's 12:22, and we just finished verification with the monitor for all transactions in one of the profiles I'm testing, with all of my partners. Day two and the second profile is done, including the HITSP testing. This one was easy because of the way the profile was written, and the small number of actors needed to complete the testing. Other profiles are more difficult to get to this point with. Time to go to lunch.

Back from lunch at 1:12. Over lunch I encouraged NIST to send some of their testing resources to HITSP meetings, just like MIR does with IHE. They are already planning to, which is a good thing. It will help us get the tests done sooner, with fewer questions.

I start looking into what's needed for my stretch goal tests.

At 1:39, I work through another problem with another engineer. When we reconfigure the system, we also need to reconfigure some message parameters. He's working on the fix.


At 1:53 we are tracking down resources back in Boston to help with a .Net permissions problem for one of the assemblies.


Ten minutes later, I'm back at my laptop still trying to get 10 solid minutes to get my stretch test started...


Two minutes into that, I get a question from another vendor about how to set up Tomcat with Certificates for ATNA testing. I point them to the ATNA FAQ on the IHE Wiki.


I checked in with the engineer reconfiguring the message parameters. He's on to the next problem.


A few minutes later, and I've had to stop again. This time to track down a networking problem with one of the systems. It isn't accepting requests from outside. I've verified that I can see it on the network. I start up Wireshark (a packet sniffer should be in the toolkit of every engineer coming to connectathon), and verify that the connection request is being immediately reset at the server end. We verify the network configuration on that system, and check all settings. The network settings were recently adjusted, so we reboot. That didn't clear the problem. Time to call technical support for the product to see if they can help.


Meanwhile, today is the day of the Connectathon conference. This is the time when C-levels in suits come to hear about what all the folks in jeans and sneakers are doing downstairs. As part of the conference, they come get a tour of the connectathon floor. The noise level just escalated by about 10-15 db as they arrive for the tour.


It's now 3:28 and I'm still working on this one test. I've made substantial progress, but since I updated some of the web services that the system used to use to get the information, I need to update some assemblies to use the new WSDL. It's just a change in the name of the message, so a few minutes later, I should be able to redeploy and restart the service.


At 3:53, I've just fixed a number of other areas where the HL7 2008 schemas is stricter about the values they permit. Some of these values had defaults in the old schema which aren't present in the new one. I've fixed my code to set the correct values, and think I'm about ready to produce the Immunization (IC) document from the QED data stream. Time to take a quick break to see how folks are doing with the networking ... Nope, it's still spelled notworking.


Checked in with one of the connectathon monitors. His work of reviewing the content for the PCC profiles is almost completely done except for some stragglers (e.g., me for one). We seem to have gotten to the point that people are understaning how the PCC Technical Framework is designed. Last year, they were still reviewing a significant content on Thursday and Friday.


At 4:41 I head back from the engineer with the notworking network; tech support found our problem. We were binding to localhost, instead of the IP address on the port. While I knew it had to be something like that, I didn't know where in the application that was configured. It's not what you know, but what you know how to find out. Suggesting that we call support was the right move. Now that I know the symptoms, I'll spot that one sooner, because I'm sure it will happen to someone else again.


I check in with our teams before the end of the day; the results are mixed. We are doing much better this year than last. That seems to be the general sense of others as well. Time for a glass of wine.

Notes from Connectathon

Today starts the week-long event known as the IHE Connectathon. This year, I thought I'd keep a little diary of how the event goes, and post it in my blog (which I've been ignoring as I prepare for this event, and run to various standards meetings).

The Connectathon is held in downtown Chicago these days. Gone are the days that we fit into the basement of the RSNA garage. Usually the connectathon occurs in Janaury in the middle of Chicago winter. We got a slight reprieve this year because of show schedules. There are more than 120 systems here, and about 350 engineers testing, many of them new to the event. Tens of thousands of transactions will be tested and verified by the end of the week.

You can tell the new people from the old by how well they know their way around. There's a lot of new people here this week. I also see a lot of old faces this week, and get some time to catch up.

After I register and get my nametag, I dig out the "Troublemaker" badge I've now recycled for four standards meetings and apply it. I immediately get rebuked by the lead security testing monitor for the event, who has more of these badges she intended to use to identify the security folks. Don't impersonate a security person I'm told. I tell her to read the badge, she laughs and we move on.

My first real step of the day:
Unload the laptop, plug it in to power and network, and log into KUDU to download the HOSTS table.

Second step:
Read the tests I'm signed up to do (I should have done this earlier, but I'm taking a few liberties due to HITSP meetings last week).

Step three:
run net time /setsntp:time-a.nist.gov and fill out details for completing my first test. The monitor came by in record time and I passed that test. One down, twenty to go.

Now it is time to reboot and actually connect to the Connectathon network using my assigned IP Address.

After an hour into testing, I've already verified connectivity with two systems, spotted some code I forgot to write to enable TLS (it's the client side, which I wasn't planning on being critical, but am now the only client for this profile). Fifteen minutes later, after reading the ATNA FAQ (which I wrote about four years ago because I have a lousy memory), TLS is enabled and working. I've verified that I can query one of the two systems I need to talk to, but still have some things to do because I didn't catch everything I should have testing with an internally developed system.

I've already seen and/or participated in the vendor-A engineer sits down to help debug vendor-B's system twice before lunch, and know it's going on elsewhere while I'm doing my own testing.

During a brief lunch break, I impersonate a security monitor (at the table where they are all sitting).

After lunch, working on a profile with someone else, we discover a need to create a change proposal to the profile we are testing to help clarify something that is obvious to the profile editor (me), but not to the reader (my testing partner). Rereading it now, it's pretty clear reading it needed to be better specified, but that's part of why we are here.

This will NEVER happen at a customer site, but after lunch I overheard a discussion between two vendors: "I talked to the guys back home, and we've identified the problem. The fact that we're sending you a (undecipherable techno-babble) is definately our fault. We'll be fixing that shortly."

I made a bunch of notes for myself before lunch. I'm going through my system to verify that I've done everything in those notes.

Later I sit down at a different system and help someone work through a problem.

While headed out to check on something, I ran into one of the two CCHIT folk who are here this week. They are acting as monitors to get an idea what real-world testing looks like at the IHE Connectathon. They'll also get a pretty good idea about how "shovel-ready" these profiles are for EHR interoperability, and for placement on the CCHIT roadmap.

The head guy in charge comes by and asks if anyone around knows anything about VMware. I speak up and head over to another system to help someone connect to the network. It's fairly common here to run applications in VMWare for testing purposes, since development and deployment environments are very different. Sometimes the configuration can get tricky.

By the end of the day, we've figured out all of our problems on the one profile amongst the three of us that are testing it, verified that we are sending the right values back, and are ready to run the test. Unfortunately for us, this test is live proctored, and the monitors have just been released. That's not a huge problem. We'll be done with this profile tomorrow, and I might even get to take on some stretch goals this week.

Tommorow is day two. Time to pass a bunch of tests. On the way out, I check in with the fellow that was having VMWare issues, and he's on to the next stage, having gotten past those.