LLP - Lower Layer Protocol

The Lower Layer Protocol (LLP), sometimes referred to as the Minimal Lower Layer Protocol (MLLP), is the absolute standard for transmitting HL7 messages via TCP/IP. Since TCP/IP is a continuous stream of bytes, the wrapping protocol (i.e. headers and trailers) is required for communications code to be able to recognize the start and the end of each message. The Lower Layer Protocol is the most common mechanism for sending unencrypted HL7 via TCP/IP over a local area network, such as those found in a hospital.

An HL7 message must be wrapped using a header and trailer (also called footer) to signify the beginning and end of a message. These headers and footers are typically non-printable characters that would not be shown in the actual content of an HL7 message.

The typical structure of an HL7 message being sent via LLP is described in the table below. It contains four parts:

Moreover, you must also ensure that each segment is terminated by an 0x0D carriage return. This is mandated by the standard, but often HL7 log data can be received via FTP or email where the segment separators have been transformed into 0x0A characters, etc.

Hybrid Lower Layer Protocol (HLLP)

The Hybrid Lower Layer Protocol (HLLP) is a variation of the more widely used Lower Layer Protocol. Like LLP, HLLP uses TCP/IP as its transport but incorporates error detection and verification via the use of checksums at the end of messages. The checksums are used to verify that no data was corrupted. Checksums are typically computed for each block of data that is sent for the sending application and then verified for accuracy at the receiving application.

In practice most vendors choose to use TCP/IP over LLP, rather than HLLP. LLP is a very simple protocol to incorporate and can be used instead of HLLP since the TCP/IP channel provides all of the services necessary for the error-free delivery of HL7 messages. This includes:

1. Connection Handshaking

• The process by which two systems initiate communication. Start and end control characters are used to start/stop the transmission of data.

2. Full Duplex Data Transfer

• The process by which a system transmits and bi-directionally receives data simultaneously.

3. Error Detection and Retransmission

• The process by which the transport layer detects segments that fail transmission and retransmits them, if necessary.

4. Flow Control

• The process by which the flow of messages between systems is managed by TCP through the use of ACKs and NACKs. Through the use of ACKs/NACKs and other built-in mechanisms in an HL7 application, you can manage the flow of data to ensure that messages are transferred efficiently and reliably.

5. Connection Termination

• The process by which a connection is ended independently by each system via a handshake.

In most cases, HLLP is an unnecessary requirement as long as the two communicating health systems are using a reliable OSI transport layer since the transmission of messages, as well as the integrity of messages, is already verified by the underlying OSI protocols.

Why is HL7 Not Plug-and-Play?

The following table explains why HL7 is not plug-and-play:

In this environment it is important to be equipped with tools that were created to deal with the diversity of HL7 implementations, standards and versions.

Overview

The structure of an HL7 message is as follows:

Message --> Segments --> Composites --> Other composites or primitive data types.

You will encounter the generic term 'field' that will usually refer to either a whole composite or to a sub-field (single value). In this document, a whole composite is referred to as a field (...|DOE^JOHN^^^^MD|...) and a single value is referred to as a sub-field (...^JOHN^...).

Messages

Let's take a look at a typical HL7 ADT^A04 message. This message is sent when a new patient arrives at the hospital. The patient's demographics are entered into a HIS (hospital information system), and then the information is communicated to all the other systems to avoid multiple entries of the patient's demographic information.

Example Message

HL7 Messages are ASCII messages (unlike protocols such as DICOM), and the standard requires that they be "human readable". This is acceptable if you don't mind counting pipe characters.

Each message consists of the segments that are delimited by "carriage return" characters ("\r" or 0x0D). Each segment is placed on a different line.

Segments

Each line in a message is referred to as a Segment, with each one having its own semantic purpose. This means that it contains information of a specific type. For example:

• MSH segment contains information about the Sender and Receiver of the message, the type of the message, a time stamp, etc.

• EVN contains information about the type of message; for example, A04 (Register a patient). Information contained in EVN is duplicated in MSH, so starting from HL7 version 2.3 this segment is excluded from all message definitions.

• PID contains demographic information about the patient such as name, id codes, address, etc.

• PV1 contains information regarding the patient's stay in the hospital, such as location assigned, referring doctor, etc.

As you can see, the sample message in the previous section contains MSH, EVN, PID, NK1 and PV1 segments. There are more then 120 segments defined in version 2.3 of the standard.

Segments consist of fields that are composites. Composites are delimited by "|" (pipe) characters. Each field has its own unique purpose and is defined by the HL7 standard for each segment.

The PID segment in HL7 version 2.3 consists of thirty composite fields. Only two of them are mandatory: Patient Name and Patient ID. As you can see, some fields consist of a single value (e.g. field 8), while the other fields consist of a number of values delimited by ^ (e.g. field 5, patient name).

Delimiter Characters

The following table lists the delimiter characters used in HL7:

Delimiter Escape Sequences

Some user data may contain these special delimiter characters. For this reason, HL7 has a system for escaping them. The system is a little bit unusual in that unlike a language like C, each character has a unique escape sequence.

The following table shows the escape sequences for each of the different characters:

You will notice that even the escape character itself has an escape sequence.

Special Escape Sequences

There are some special escape sequences that are validated and passed through without escaping or unescaping:

Delimiter Redefinitions

If you are new to HL7, one thing that can be really confusing is the fact that delimiter characters can be redefined on the fly in the MSH segment of each message.

Each HL7 message starts with a MSH segment, much like the following:

MSH|^~\&|...^...|

It is at this point in the message that some HL7 implementations choose to use different delimiter characters. For example:

MSH$%#!($...%...$

Example Message Definition

An example of a message definition is as follows:

MSH PID PV1

The example above means that the message must contain only MSH, PID and PV1 segments in this order.

Optional Segments

When a segment is optional it appears in square brackets. For example, the message MSH PID PV1 [PD1] indicates that MSH, PID and PV1 segments must appear in the message in this order, and a PD1 segment may appear, as indicated by its enclosed square brackets, after PV1.

Repeating Segments

When a segment may repeat, it is enclosed in curly brackets, like {NTE}. It means that at least one appearance of the segment must be present, and it can repeat any number of times.

HL7 Segments Summary

MSH PID PV1 [PD1] {NTE}

• The above indicates that MSH, PID and PV1 segments must appear in the message in this order, and a PD1 segment may appear after PV1, and NTE (Notes and Comments) must appear 1 to N times at the end of the message.

Segments that are both Optional and Repeating

• A segment that is both repeating and optional is enclosed in both square and curly brackets, like[{NTE}].

Groups of Segments

• As described above, entire groups of segments may be present as either repeating, optional or both. In this case, the segments that the group consists are enclosed in the appropriate brackets.

Example

• Consider a patient information group in some fictitious ORU message. It consists of a mandatory PID segment, optional PD1 segment and repeating optional NTE segment. So HL7 notation for this group is PID [PD1] [{NTE}]. The message itself can contain information about any number of patients from 0 to N, so this group is itself optional and repeating. Notation for this message is MSH [{ PID [PD1] [{NTE}] }].

Some familiarity with the notation might be useful if you are reading an interface description provided by a vendor that you have to interface with.

Z-Segments

While developing the standard, the HL7 committee realized it was impossible to standardize all the aspects of data exchange in the healthcare industry. This is why they provided a framework for the addition of custom information into HL7 messages, more commonly referred to as Z-segments. You may define a custom segment and include it in any HL7 message. These segments should be named starting with letter Z, (e.g. ZPD for custom patient demographics information). Applications that do not know the meaning of your Z-segments should simply ignore these segments.

The downside of the Z-segment approach is that you cannot expect any particular piece of information to be in a specific place. While one vendor will send this information in some segment of the standard, another will send it in a custom Z-segment.

When selecting a toolkit to deal with HL7 integration, it is extremely important that your code be shielded from the actual message structure.

When you refer to a field such as PID.ExternalId in your code, you will have to change your code if this piece of information begins to appear instead in a ZPD segment.

Present but Null Fields

One question that arises in HL7 quite often is how to tell if a field is present but null in an HL7 message. When a field is absent from a message it should mean that the system does not use that field, or that the field has not changed. However, what if a system does support the given field, but the value of the field is null? The HL7 protocol requires a method of making this clear.

To make present but null fields clear, you must put two double-quote characters together in a field like this:

OBR|||||""||||

This is a very common technique used in many real-world implementations.

MSH|^~\&|EPIC|EPICADT|SMS|SMSADT|199912271408|CHARRIS|ADT^A04|1817457|D|2.3|
EVN|A04|199912271408|||CHARRIS
PID||0493575^^^2^ID 1|454721||DOE^JOHN^^^^|DOE^JOHN^^^^|19480203|M||B|254 E238ST^^EUCLID^OH^44123^USA||(216)731-4359|||M|NON|400003403~1129086|999-|
NK1||CONROY^MARI^^^^|SPO||(216)731-4359||EC|||||||||||||||||||||||||||
NK1||DOE^JOHN ^^^^|SPO||(216)731-4222||EC|||||||||||||||||||||||||||
NK1||DOE^ROBERT ^^^^|SPO||(216)731-4222||EC|||||||||||||||||||||||||||
PV1||O|168   ~219~C~PMA^^^^^^^^^||||277^ALLEN FADZL^BONNIE^^^^|||||||||| ||2688684|||||||||||||||||||||||||199912271408||||||002376853

Group of Segments

A group of segments is a collection of segments that can be treated like a single segment for repetition purposes. For instance, a group of segments may be optional as a group and may repeat as a group.

Consider the following example:

If an ORU^R01 message has a group that contains one OBR (Observation request) and 0 to N OBX (Observation result) segments, this group is optional in the message so it may not appear at all. The message will then appear like this:

MSH PID PD1

A message may also contain a number of observation groups. For instance, when the OBR-OBX group repeats, the message appears as follows:

MSH PID PD1 OBR OBX OBX OBX OBR OBX OBX

The first three observation results belong to the first observation request, and the next two observation results belong to the second observation request:

MSH PID PD1 - header and patient demographics OBR OBX OBX OBX - first observation group (e.g. height and weight) OBR OBX OBX - second observation group (e.g. lab results)

If you are going to parse a message that contains groups of segments, you need to create a complex structure similar to relational database tables. This will allow you to retain a relationship between the segments in the group, rather than a long array of segments.

ACK Protocol

An important part of the HL7 standard is the ACKnowledgment protocol. Every time an application accepts a message and consumes the data, it is expected to send an ACKnowledgment message back to the sending application. The sending application is expected to keep on sending a message until it has received an ACK message.

If you do not follow this rule then data may be lost in the transmission. If you always follow it then you will never lose HL7 data.

Message Control ID

The key concept in the ACK protocol is the Message Control ID. This is a unique number that every HL7 message has in field 10 of its MSH segment. A valid HL7 ACK will echo this ID back in the second field of its MSA segment. This diagram show a typical ACK message with the important fields labelled:

ACK Message

You will notice that it consists of two segments:

• MSH segment

• MSA segment

MSH Segments

The MSH segment is the MeSsage Header segment. Every HL7 message starts with an MSH segment. This segment has the header information about the sending and receiving application and facility. It also contains the version of each message. More importantly, it has the Message Control ID of the message, which is the unique ID of each message. For the inbound parsing of each message you want to acknowledge, it is important to store the message control ID and possibly other parameters like the sender, version etc.

MSA Segments

The MSA segment needs to have the message control ID of the message being ACKnowledged in the second field. The first field should be a constant, AA, meaning a positive acknowledgement without errors.

Business Processes for Software Vendor Customers

The roles and processes documented here can be leveraged by our Software Vendor customers to better plan and manage their business with respect to HL7 interfacing:

Primary Integration Engineer

The Primary Integration Engineer's main responsibility is to set up the initial infrastructure and implement a point-of-care (POC) interface. They are also responsible for ensuring that either Chameleon or Iguana has the necessary infrastructure to work in their environment.

If the customer is a Software Vendor, this role includes integrating the vendor's product with Chameleon and/or Iguana to allow the Field Integration Engineers to perform their site by site integrations. The Primary Integration Engineer needs deeper technical skills than the Field Integration Engineers but does not necessarily need the same degree of people skills. Ideally they should be expected to do some of the site integrations themselves (especially the initial one) using the infrastructure they have developed to get a feel for how the system is used by the Field Integration Engineers.

The effort required from this role depends on the interfacing product being used:

Field Integration Engineer

The Field Integration Engineer is responsible for implementing each site's deployments and customizations. They leverage the groundwork already established by the Primary Integration Engineer. This usually involves basic customization of a template VMD file already established so that it is tailored to generate and/or parse the site specific dialect of HL7.

A solid understanding of the Chameleon Interface Development Environment (IDE) is necessary for this role, since - in the worst case - a good deal of VMD file customization may need to be carried out. This includes not only drag-and-drop modifications to segment grammars, table grammars and mappings, but also modifications to Python scripting where applicable. As such, the training of the Field Integration Engineers is a key factor in determining the overall efficiency of the HL7 interfacing operation. If these engineers are not trained effectively then this will drive up the cost of HL7 interface implementation and ongoing maintenance.

Skills Required

• Solid understanding of the Chameleon Integrated Development Environment (IDE)

• Basic knowledge of HL7 (e.g. the contents of our manual's HL7 Overview chapter)

• Understanding of how to use the basic iNTERFACEWARE HL7 Analytical Tools for testing interfaces: the Simulator, Listener, msgdiff and msgtransform tools

• Competent at writing good python code to implement scripting required for different site customizations

• Basic knowledge of TCP/IP communications, where applicable

• Good communication and people skills required to interact with site technical people

Quality Assurance Engineer

The Quality Assurance Engineer is responsible for maintaining the quality of the interfacing effort on an ongoing basis. One of their primary functions is making sure that test environments are installed and that upgrades to either Chameleon or Iguana are carried out smoothly. They are responsible for ensuring that the core messaging infrastructure established initially by the Primary Integration Engineeris routinely regression tested (this is especially critical when a pure Chameleon SDK approach has been adopted).

It is usually best to keep this role separate from the Primary Integration Engineer role as this is an ongoing operational role, whereas the Primary Integration Engineer role is generally not. This role can, however, be satisfactorily merged with the Trainer role.

Skills Required

They need to have experience in standard quality assurance methodologies and test automation and should be familiar with the quality assurance suggestions outlined in this guide.

Accounting/Contract Personnel

The role of the accounting personnel with respect to integration is twofold:

• To efficiently ensure that a complete list of sites that have HL7 interfaces is maintained. iNTERFACEWARE recommends that the customer maintain a Site Database for this purpose.

• To ensure that valid runtime licenses are issued for each site that needs one. It is recommended that Purchase Orders to iNTERFACEWARE are consolidated into orders once a month to minimize administrative overhead.

Ensuring Quality of the Core Infrastructure

Recall from the explanation of the Primary Integration Engineer role that the core infrastructure work required to support a pure Chameleon SDK approach to interfacing is much higher than the work required if an Iguana approach is adopted. It naturally follows, then, that the effort required to do the quality assurance of a Chameleon-based core implementation is similarly much higher in comparison to an Iguana-based solution.

• With an Iguana-based solution the message delivery (networking, batch file, etc.), logging, administration, database access, etc. has been developed and QAed by iNTERFACEWARE. Not only has it been developed by iNTERFACEWARE, but it is deployed to hundreds of sites worldwide. As such, the only aspect of the core messaging infrastructure that requires QA by your team is your application's basic interaction with Iguana (i.e. the shared database or the interface with the client executable).

• With a Chameleon-SDK-based solution, however, there is much more core infrastructure that needs to be QAed. The message delivery, logging, administration, database access is all custom code implemented by the Primary Integration Engineer in this case. As such, it is absolutely critical to carry out thorough QA on all of these aspects of the core implementation. If networking code has been implemented to deliver messages via the Lower Layer Protocol over TCP/IP, for example, a thorough checking for all possible networking issues must be carried out. Some checklists to be aware of for a Chameleon-SDK-based implementation include:

• Red Flag Quality Issues

• HL7 Interface Network Issues

Red Flag Quality Issues

This a basic checklist of red flags to look for with a custom interfacing implementation built around the Chameleon SDK. Most of this list does not apply to Iguana implementations since Iguana naturally avoids all of these common mistakes.

No functional specification?

• It is important to avoid beginning implementation until a basic functional specification has been written up. This is important because it is not uncommon for the Primary Integration Engineer to get carried away with thinking about low level details of the implementation (i.e. how to solve the problem rather than 'what is the problem?'). As a result, many customers go through an expensive re-implementation of the first cut of their core messaging infrastructure because they have produced an application that is inconvenient and impractical for the Field Integration Engineers to use. A good resource on functional spec writing that explains why it's important is:

• http://www.joelonsoftware.com/articles/fog0000000036.html

Site-specific business logic implemented in code?

• Chameleon and Iguana are designed to make dealing with non-standard flavors of HL7 easy by allowing customers to define site-specific configurations with different mappings and python manipulation at each site. All of this logic is nicely encapsulated in an interface configuration file, or "VMD file".

• Sometimes engineers will have instead implemented this business logic in the language they are writing their core infrastructure implementation in such as Java, C#, C++, etc. This is very bad practice since it makes supporting different flavors of HL7 very difficult, dramatically driving up the cost of deployment for each new site.

A different application per site?

• This is a huge quality problem. Here the Primary Integration Engineer will quickly create a simple application for each customer site based directly off of the generated client/server example code. This is a maintenance nightmare. The core infrastructure code needs to be thoroughly thought out and tested. This development takes time but should only happen once, up front. New site deployments should not require any changes to the core application code. For that reason, consider using the multiple configurations functionality to deploy variations of the same VMD file at different sites. Not only does it reduce the cost of supporting various flavors of HL7, it also reduces the overall maintenance effort and keeps your core infrastructure code base stable.

Errors Logs not clearly separated from normal log messages?

• Specific error conditions should be easy to spot and filter from normal logging activity in an HL7 interface. Having a clear time line is very important, too, with timestamps for when activity occurs.

• A good quality implementation will make it very easy to drill into and review the logs without requiring any deep technical knowledge.

• Iguana separates normal operational logs from error logs and gives users the ability to search those logs through its web interface.

No easy means to export the HL7 messages being sent and/or received?

• It is important to have an easy means for exporting messages being sent and received for audit purposes, quality assurance, and for maintaining a detailed site database.

• Iguana has the ability to export logs of processed messages through the web interface.

No log purging in place?

• One common mistake that is made with HL7 interfaces is to forget about what happens when the logs fill up and overflow the system's hard drive. A good system will have some method of truncating logs so that they are periodically purged.

• Iguana has log purging built in so it's easy to configure. If your machine's hard drive is small, consider purging your logs 2-3 times per week to allow Iguana to run optimally.

Is a complete reboot required for configuration changes?

• It is easier to write servers that require the user to restart the entire server to pick up configuration changes. In practice for the users - i.e. the Field Integration Engineers will appreciate a system that does not require rebooting to pick up configuration changes. It gives much greater comfort to your customers who are unlikely to appreciate interfaces going up and down.

• Iguana allows channels to be added and removed on the fly without needing to restart it or turn other channels off and on.

No clear alert strategy in place?

• A good alert strategy will ensure that a Field Integration Engineer is paged in the event of a serious problem with an interface.

• Iguana has the ability to send email alerts when errors occur with the HL7 interface.

No monitoring system in place?

• Is there an easy way to get visibility into the functioning of the system. Preferably this interface should not require deep technical skills for the person doing the monitoring so that it is like Iguana which has a nice web based interface to browse the logs.

• Like Iguana there should be permission levels which control what each user can do with the system.

Is the VMD file being loaded with every message?

• This is a fairly common mistake. Loading a VMD file takes a reasonable number of seconds which means that if a Chameleon application is loading the VMD every time it parses a message it will result in very poor performance. A VMD file should only be loaded once or when it changes.

• An adequate HL7 interface should be happy handling up to 6-7 messages per second - this reflects the volume one would expect for an ADT interface with 60,000 messages per day where 80% of the load is between 9-10 am and 4:30-5:30pm.

Is it possible to configure a set of filters in an interface?

• Chameleon has the ability to do Scripted Transformation, which takes an HL7 message and transforms it into another dialect of HL7. This can be a very helpful tool in terms of making site-specific customizations. A good implementation will allow a Field Integration Engineer to leverage this functionality in Chameleon.

• This is functionality often overlooked by Primary Integration Engineer when implementing custom solutions with Chameleon without Iguana.

Does the application handle badly formed HL7 messages?

• Invalidly formed HL7 messages being fed to the server should not result in the server crashing. It should be able to handle such data gracefully.

Does the server leak memory or resources over time?

• Languages like C# and Java supposedly make memory leaks impossible. In practice, however, even these languages can leak memory through circular references and all languages can leak resources like database handles, etc.

Does the application crash when a database it depends on disconnects?

• This is a standard thing to check. Poor implementations will often statically link into database implementations making them impossible to even run if the appropriate dependencies are not there. This should be avoided.

Does the application handle receiving unknown HL7 messages?

• The trick here is that the application should be able to just acknowledge and log the unknown message. This makes the interface more stable when extra messages are sent to it and prevents application errors when unexpected data are received. Chameleon does allow one to define a default message - the feature is very obvious in the 4.1 GUI but was more difficult to locate in earlier versions.

Does the application have sufficient functionality to handle ACKs when sending messages?

• There are four common scenarios to support (Iguana supports them all):

• In well behaved sites it is important to match the message control ID coming back in the ACK. The application should allow configuration of this logic like Iguana does with a VMD file like ack_verify.vmd to parse out the message control ID.

• Some sites will just send back ACKs where the message control ID does not match - so the solution will need to have the ability to accept any message as an ACK.

• Some sites will send back a message but not validly formatted as an ACK - i.e. it might be three letters like "ACK".

• Some sites are really badly behaved so that one needs to be able to configure a sender to just send messages and not wait for ACK messages.

Does the application support sending NACKs?

• Generally for implementing an HL7 listener to receive messages there will be two types of behavior required when errors occur:

• With regards to message processing errors, some sites will prefer that a negative ACK (NACK) message is sent. This should be configurable using a VMD file, like autonack.vmd, for setting up negative acknowledgements.

• In some sites sending a NACK will stop this interface. So sending NACKs should be disabled.

• Iguana allows these options to be configured.

Do the interfaces fail to start up smoothly when the machine is rebooted?

• Just a standard test to do. HL7 interfaces should be written as service applications. If there are dependencies on other services, like databases, then care should be taken to ensure that these services start up before the HL7 interface application starts or that the HL7 application can handle the database connection not being available initially.

Does the application support inputs and outputs other than just HL7 via LLP over TCP/IP?

• About 10-15 percent of HL7 interfaces require other transport mediums like batch files, etc.

• Iguana supports input from batch files, databases, etc. Also, the client executable interface means it's easy to implement custom transports if required like web services, etc. without the need to write an entire application and re-release it.

Can the application be patched easily?

• Iguana's client executable support makes it very easy to add new functionality specific to a single site without needing to re-release the entire application. The reason is that each client executable is loosely coupled to both Iguana and to the other part of the channel, whether it is a built-in component or another client executable.

• So if you encountered the situation in for one customer site where say a custom transport was required like setting up a web service, then existing production code does not need to be altered. Instead, a new client executable module can be implemented to handle this requirement. This avoids the need for an entire release and QA cycle on the existing production code.

• Many in-house implementations using a raw Chameleon SDK approach will not necessarily have been factored to allow this flexibility. Business logic will often be mixed in with transport and HL7 generation logic. Often complex low level threading architecture issues interfere with making what should seem a simple new requirement. It takes a very disciplined approach on the part of the Primary Integration Engineer to implement this type of modular breakdown.

Is deploying the application difficult?

• An automated setup file should be made or simple step-by-step instructions should be provided for the Field Integration Engineers to follow.

• However, make sure that Chameleon is installed and unaltered in its entire form with the same version that you developed with. The installation of Chameleon can be automated, which makes this process easy. Many serious problems can be created when software engineers try to make a custom deployment of Chameleon by copying DLLs in Windows directories, etc. This will often result in "DLL hell". Best practice is to install the custom application into the Chameleon directory so that it loads all the correct library versions.

HL7 Interface Network Issues

When implementing an interfacing solution directly using the Chameleon SDK, there are a number of things to consider, specifically with regards to message delivery over TCP/IP. This page does not apply to Iguana implementations since Iguana handles all the networking side of HL7 interfacing and has been tested under a wide range of hostile TCP/IP conditions.

Writing good quality network interface code that can handle various problems with TCP/IP is not trivial. It is important that this is realized by the system's Primary Integration Engineer.

The most common mistake is to only test with well behaved TCP/IP listeners during development and not anticipate how some counterparties in production will not be well behaved. Typically this is because many HL7 interfaces that you will encounter in production have not been implemented well by other Primary Integration Engineers that also did not know about the issues with networking problems.

In an ideal world the following would be the way that all outgoing HL7 interfaces, for example, would work:

• 1. Poll application for message.

• 2. Send message.

• 3. Wait for an ACK.

• 4. Either:

• Receive ACK, verify that ACK is well formed and the message ID matches message sent. Then, mark message and sent in database and go back to step 1.

• Time out. This should be configurable. Resend the message and go back to step 3.

In practice, however, a number of complications enter into the above scenario. One of the biggest problems in TCP/IP network programming is that everything is asynchronous (i.e. even though the application may think it is connected to the remote host, it may not be since the remote host can disconnect at any time.

Well-written networking software will always check every single network operation and gracefully recover when problems crop up with the network - like when the counterparty disconnects. Poorly written networking software is what we call 'Good news code' which assumes everything is working as expected. 'Good news code' is really bad news since it leads to intermittent, mysterious and difficult-to-troubleshoot problems on customer sites.

A QA person will need to write a program to simulate a hostile TCP/IP counterparty to rigorously check an in-house written HL7 client server to ensure that it can reliably deliver messages without losing data to a poorly behaved HL7 server. In terms of effort, about 2-3 days of development time for the QA person should be budgeted for every day of development.

Other common networking problems to test are:

• Test with high latency. Set up a remote host on a very far away server. Using a remotely hosted web server is not a bad way to do this. Obviously some custom software will need to be written to do this.

• The application should be configurable to work with some systems that cannot be guaranteed to send back valid ACKs for each message.

Bringing a New Site Online

Hospital sites will vary greatly in their level of preparedness and organization. One of the key things in preparing a site is getting the contract in place with the hospital that will cover key points in the relationship. Namely:

• How remote access will be given. Hospitals will sometimes be reluctant to give remote access. One way to negotiate this will be to give the option for it and charge extra if the hospital does not provide remote access.

• How the hospital will pay to have the site maintained. Maintenance is a vital part of HL7 interfacing since things will change over time at each site. The contract should make it clear that if the hospital changes the software systems in their environment in a manner that requires your organization to implement interface changes then this cost will be billed.

• How faults at each site will be raised to the attention of your Field Integration Engineers.

• The customer should be required to sign off on the HL7 specifications before the implementation begins. This is important to prevent unpaid change requests from being made.

Additionally it is very desirable if some of the information needed for the site database can be collected during this phase. Most critically, you will want to collect a set of sample data from each of the systems you will be communicating with. In practice this may prove to be quite challenging to orchestrate; however, clear and consistent requests can make all the difference.

Once the PO has been received with the contract in place, the process for bringing a new HL7 site on line can be broken into the following steps:

• The Accounting/Contract Personnel should add the new site to the list of sites that require runtime licensing for the month.

• The Field Integration Engineer should do the integration implementation work by customizing a base template VMD file. They can use a temporary or permanent product license to carry out the implementation.

• If a temporary license is used during the implementation, the Field Integration Engineer should notify the Accounting/Contract Personnel upon completion of the integration. When payment is received from the customer, the Field Integration Engineer should install a non-expiring license key.

• The Quality Assurance Engineer should receive the data and VMD files from the Field Integration Engineer for the new site and carry out the standard QA Processes for a New Site.

Ensuring Quality for a New Site

One key step in the quality assurance process is to ensure that the HL7 database has been filled out methodically for every site.

The quality assurance effort required on a site-by-site basis should be far less if the groundwork has been done by Ensuring Quality of Core Infrastructure. If that is done well then most of the validation required for each site should be possible to do just using Chameleon and the command line msgtransform andmsgdiff tools.

This is because all the site specific customization for each site should only occur within the VMD file for that site; meaning, only the way that mappings and python scripting is done changes from site to site.

Validating Outbound Interfaces

Ideally you should be getting less variation on outgoing interfaces compared to incoming interfaces. See the HL7 Specification resource.

For validating outgoing interfaces, a process will be required, similar to the one below:

• There should be a sample of a few hundred HL7 messages in the format expected by that site - i.e. the expected output.

• There should be a simple way of populating your application's database with a set of test data to produce that expected output - i.e. the standard input.

• The resulting output can be compared to the expected output using the msgdiff tool.

As the application grows and improves the standard input may change - but the structure of the expected output should be the same. The software implemented by the Primary Integration Engineershould support this quality assurance process.

Validating Inbound Interfaces

This is easy to automate so long as all the site specific customizations are implemented within the VMD using configurations and python scripting. For each site the sample data (i.e. HL7 messages) can be stored and then run through the VMD file using the msgtransform tool to validate the behavior - i.e. standard input and expected output.

This can be automated with scripting. A batch file can be used to perform automated regression tests that will allow you to compare changes in output rapidly.

General Checks

• Python scripting in each VMD file should in general be easy to follow and commented where necessary.

• The VMD files for each site should be stored in the database.

• If VMD files are customized on site then a VMD merge tool should be used to combine the site customized VMD files into a single master VMD file. This merge tool can be obtained from iNTERFACEWARE.

• Use multiple configurations to deploy variations of the same VMD file at different sites, which reduces the maintenance effort and upgrading costs.

Upgrading a Site

If the guidelines for Ensuring Quality for a New Site have been followed and the HL7 database has been maintained correctly it makes upgrading sites to new versions of the HL7 interface much easier.

The Primary Integration Engineer should repeat the steps for validating interfaces taken for quality assurance in a new site. They should then take an up-to-date HL7 data sample for the current site and run it through the msgtransform tool to have another set of data to compare with the new HL7 interface and new VMD file. This should help identify problems early on before deploying the new site.

Ideally it should be possible to deploy the new software on a test server before altering the production server. There should also be rollback procedures to restore the earlier version of the software if necessary.

Accounting Processes

The following sections describe functions of the Accounting/Contract Personnel role.

Ordering New Runtime Licenses

Typically the paper processes associated with processing a single invoice can run at up to as much at $25 per invoice in a typical corporation. If an organization is making more than one invoice a month it usually saves money to consolidate those into a single monthly invoice.

Combined with a good site database this makes for an orderly and efficient way to reduce paperwork overhead.

Maintenance Collection

This is where the HL7 database is invaluable in keeping track of which sites should be billed. Maintenance on Chameleon and Iguana runtime licenses should only be paid on those sites that are actively paying for maintenance.

In larger organizations the HL7 database's function will normally be integrated into a larger CRM application.

Key Information

The following information should be recorded in the site database:

1. The number of outgoing interfaces at each site. For each outgoing interface the following information should be recorded:

• IP address/host and port number to send to.

• System name to send to - e.g. Meditech, Cerner Millenium, Star HBOC, etc.

• Types of messages being sent - e.g. ADT^A01, etc.

• Typical daily volume of messages.

2. The number of incoming interfaces at each site. For each incoming interface the following information should be recorded:

• Port number.

• System name receiving from - e.g. Meditech, Cerner Millenium, Star HBOC, etc.

• Types of messages supported - e.g. ADT^A01, etc.

• Typical daily volume of messages.

3. Large sample of real HL7 data for the site. One issue with storing real HL7 data are the privacy issues associated with HIPAA.

There are three possible solutions:

• Restrict who has access to the data within the organization.

• Use Chameleon with scripted transformation to scrub confidential data from these HL7 samples.

• Try and only have test data (as opposed to real data) from each site. Some sites will be organized in this area. Others will not or have test data which is inconsistent with their real live data.

4. Do they have a maintenance contract? Highly recommended.

5. A list of any site-specific issues should be maintained for future reference. It's nice if this database can be integrated with an email database.

6. Main interface contact person for the site.

7. Remote access procedures and information on the upgrade policy of the site. Some larger hospitals can make this difficult.

• It will greatly reduce the costs for an organization if a standard means of remotely accessing each site can be developed. It is a good idea to make this part of the contract for a customer to buy an HL7 interface - if the customer does not then give remote access or requires a non standard means of doing so then the contract should be default require extra payment to cover the additional cost in time.

8. The VMD files associated with this site.

HL7 Interface Specification and Contract

Having a good HL7 specification document can be an extremely powerful tool for doing business with your hospital customers. One thing that it is very helpful is to try and prevent your customers from pushing unwanted time and effort on to your Primary Integration Engineers through uncontrolled change requests.

With respect to the outgoing streams of data the basic rule of thumb is to adhere as closely to the standard as possible and try not to be too flexible.

With incoming streams of data the use of Chameleon makes it easier to be very flexible in what kind of format the data comes in.

As a result the HL7 specification document you produce should be extremely detailed about what format of HL7 your application produces and how it is structured (note: you can use Chameleon's documentation capabilities to generate the skeleton of this document for you). Early in the contract process the customer should be encouraged to sign off on it so that later on they are not able to request changes without a change request being issued which results in billable time.

For the incoming data, the specification can be more general since it is usually quite easy to take the streams 'as is' and account for variations by adjusting the interface configuration file ("VMD file") using the Chameleon IDE.