XDS Test Management
Generic Testing Procedures
Generic Testing Procedures Using xdstest
- Install and configure XDS Toolkit
- Review general xdstest instructions here
- Review general discussion of test from here. This page holds a description of each test indexed by test number.
- Review detailed instructions for test. They are available in XDS Toolkit:
- Open the Pre-Connectathon Tests tab
- Select Actor Name
- Select Test
- Detailed instructions are displayed.
- Toolkit will display pass/fail. Failure is in red with details. You can use the Inspect Results button to investigate failures.
- Report the results using guidance from here
Generic Testing Procedures Using Public Registry Server
- Review configuration information for the Public Registry server found here.
- Review instructions for test from here. This page holds a description of each test indexed by test number.
- Obtain a Patient ID and Unique IDs as necessary
- Run the test steps
- Report the results using guidance from here
Public Registry Testing Services
The following services are available on the Public Registry:
- Document Repository. This is a full implementation of the XDS.b Document Repository actor.
- Document Registry. This is a full implementation of the XDS.b Document Registry actor.
- Test Log facility. This captures all activity sent to the above two services. The next section describes how this facility is used in testing.
- Tools page. This allows a manual way to allocate a Patient ID for the Public Registry.
Tests for XDS and related profiles come in two forms and are supported by two testing tools. For testing your server (IHE actor implementation accepts connections from a client), a test client called xdstest is used. For testing your client (IHE actor implementation generates connections to a server) a server called the Public Registry is used.
Server Testing Background
Most of the information in this section is beyond what a user needs to know. It can be ignored.
For server testing with XDS Toolkit, there are two necessary parts. The Toolkit itself and testkit, a collection of tests that can be run. Testkit is now packaged inside the XDS Toolkit. The testkit generally contains individual tests named by a test number (11765 as an example). These test numbers correlate with the test numbers in Gazelle Test Management, the IHE test management tool which assigns tests and accepts test results.
In the testkit each test number is a directory. In its simplest form, each test directory contains a testplan.xml file and some supporting files. The testplan.xml is a test script and the xdstest tool is the script interpreter. Instructions on how to use xdstest can be found here. More complicated tests contain sub-directories named 'submit', 'eval' etc. These represent individual steps you are likely to want to run separately. Each sub-directory contains a testplan.xml file possibly with other supporing files. Each test directory contains a readme.txt file which describes the test and its individual steps. Another file, index.idx, is a text file that lists the order the sub-directory sub-tests are to be run. The xdstest tool can be run each individual test step separately or can run the entire test at once. It uses the index.idx file to know what order to run the sub-tests.
Each testplan.xml file contains one ore more <TestStep/> elements which are the atomic actions to be performed. Again like the sub-tests, the test actions can be run all together or one at a time.
The execution of a testplan.xml file results in a log.xml file. Whether you run one test step or all test steps, a single log.xml file is generated.
The xdstest tool is part of a growing collection of command lines tools referred to as the xdstoolkit. When you download xdstest you are actually downloading the entire xdstoolkit.
Tests are now managed in collections based on the transaction being tested. See the testkit/collections directory. A new option on xdstest (-tc) can be used to point the tool to a collection of tests to be run.
Unlike the operation of xdstest in prior years, there is no longer any need to edit testplan.xml files. This is because the following information is configured into the tool and its support files:
- The Patient ID for testing
- Unique ID generation (as before)
- Web Service Endpoint.
Evaluating the test
This section is no longer needed and can be ignored. Log management is now handled entirely within XDS Toolkit.
The log.xml files generated can be inspected for debugging information as needed. Each test step results in a section of the file documenting is operation. Each test step is labeled with an attribute, status, indicating the pass/fail status of the test. The top most element also has a status attribute indicating the results of the entire test event. If the top most element has status=Failure then one or more test steps has status=Failure.
All XDS metadata included in the testkit is in ebRIM 2.1 format (XDS.a). The tool converts this to ebRIM 3.0 (XDS.b) format as needed. To see the actual metadata sent, look inside log.xml at the test step of interest. Under the test step will be an element with a name ending in Transaction (RegistryTransaction is an example). Inside that element is an element InputMetadata. This contains the actual metadata sent in the transaction. Other sections inside the *Transaction section include AssignedPatientId, AssignedUids, AssignedUuids (some tests assign UUIDs while other let the Registry actor do it), OutHeader (output SOAP header), InHeader (returned SOAP header), and Result (the ebRS and ebRIM content of the response message).
When requesting assistance with testing issues, please include the log.xml file in your correspondence.
To test your client, Document Source or Document Consumer for example, the Public Registry is used. This is an Internet server that is available 12 months a year and supports many testing events. The URLs/Endpoints used are specialized for each testing event. The server configuration is documented here.
Your client, Document Source or Document Consumer for example, must manage Patient IDs and UniqueIds as a normal part of its operation. A Document Source implementation must have access to a Patient Id management module (probably with access to an ADT system). When testing against the Public Registry, a web based Patient ID allocator here must be used. This not only allocates a Patient Id but also registers it so the Document Registry implementation on the Public Registry server will accept it.
Most tests can be run using a Patient ID that you have used for other tests. If new 'empty' Patient ID is needed for a test (no submissions registered against it in the Public Registry) then the test documentation will indicate so.
For the normal operation of your Document Source actor implementation, it must allocate UniqueIds for many XDS metadata objects. No tooling is available to assist with this but the Document Registry implementation on the Public Registry does check that these ids are truly unique.
Configure Public Registry Endpoint
When a client sends to the Public Registry server, the correct WebService Endpoint must be used. Endpoints are organized by events. Each event uses different endpoints. This allows different configurations, like authentication certificates, to be established for each event. In the test documentation an endpoint is documented as
There are several things to note about this endpoint:
- PORT is a placeholder for the actual port number
- EVENT is a placeholder for the actual event code
- test11730 is a specialty endpoint for validating the results of this test. Both specialty endpoints and general purpose endpoints are offered. The general purpose ones operate per the profile specification. The specialty versions add test specific validations.
Look here to find the values for PORT and EVENT to use.
Server and Client Testing
Some actors act as both client and server. The XDS Document Repository actor fits this description. It accepts messages from a Document Source actor and sends messages to a Document Registry actor. For testing Document Repositories, xdstest is used to generate the message to the Document Repository. The Document Repository is configured to send to the Document Registry available on the Public Registry server. This way both sides of the Document Repository under test can be monitored.
This section can be ignored. The tool xdstest is obsolete and no longer supported. The XDS Toolkit (GUI) replaces it
While the most detailed (and most up to date) documentation on xdstest is gotten by executing xdstest:
these notes will help you get started.
Basically, xdstest operates on a testplan.xml file (a script file for xdstest) in the current copy of the testkit. The most frequent invocation is:
xdstest -run -err -t 11890
This call does the following:
- Finds the test 11890 in the testkit (-t 11890). The testkit is found via an environment variable established when the testkit is installed.
- Finds the log directory to deposit the log.xml file. This directory is found via environment variable established when the testkit is installed. The log directory follows the directory structure of the testkit.
- Runs the test (-run). The log.xml file is deposited into the log directory.
- At the end of the run, an error summary is displayed (-err).
Also in the background the following things happened:
- The current Patient ID was pulled from the configuration area.
- If the test plan called for a new Patient ID to be allocated then the Public Registry was called (via a Web Service) to allocate a new Patient ID. Since the allocation came from the Public Registry if this test is for a Document Repository that sends metadata to the Document Registry actor on the Public Registry the ADT Patient ID registration is already handled.
- New Unique IDs were allocated as needed.
- Endpoints for the underlying Web Service were taken from the configuration. The above invocation did not specify a site (my term for machine hosting a bunch of services). The default site was used. The file actors.xml in the configuration area was referenced. Documentation inside that file describes how to add new sites (like the actors you want to test), how to set the default site, and the WS Endpoints for each transaction on that site. It also includes the pointer to the Public Registry for Patient ID allocation. Each site has its own allocator but you are welcome to use the Public Registry server.
Basically, once the testkit is installed and xdstest is configured, you can forget about the testkit. Well, that is not quite true. Each test directory contains a readme.txt file that gives detailed directions for running the test. So, you can almost forget where you installed it.
How to report results
We would like you to upload evidence of successful results of your pre-Connectathon tests to show that you are making good progress with the tools. Note that we no longer grade individual pre-Connectathon tests, so you may choose to upload a log file or screen shot to demonstrate that you passed the test. Alternatively, you may upload a simple file that says "The test was run successfully."
If you fail to prepare your systems during pre-Connectathon testing, then your testing partners at Connectathon will reinforce how important this phase of testing is.
Some minimal test data is kept in the Public Registry server Document Repository and Document Registry actors.
- Test dataset 1
- Generated by test 12327
- This data was created using the Register Document Set.b transaction. The metadata does not point to real documents. It is good only for testing the Stored Query transaction (Retrieve of the documents will fail).
- Patient ID is daac09bb0a3f45e^^^&126.96.36.199.4.1.21367.2005.13.20.3000&ISO
- Dataset contains two SubmissionSets each with a single DocumentEntry:
- Submission 1
- SubmissionSet (id=urn:uuid:2a692702-6a1d-4c36-be54-f24acb5e7f30) (uniqueId=1.42.20141105202336.33)
- DocumentEntry (id=urn:uuid:f9ce9a0e-b2a3-4138-a309-a9ad297280ed) (uniqueId=1.42.20141105202336.32)
- Submission 2
- SubmissionSet (id=urn:uuid:574013c1-b74a-407d-ace5-ebfe799a23a5) (uniqueId=1.42.20141105202336.36)
- DocumentEntry (id=urn:uuid:e5340a2c-361f-4653-a513-bca59726ae05) (uniqueId=1.42.20141105202336.35)
- Submission 1
- The DocumentEntry in SubmissionSet 2 replaces the DocumentEntry in SubmissionSet 1
- Test dataset 2
- Generated by test 12345
- This data was created using the Provide and Register Document Set.b transaction. The metadata points to real documents which can be retrieved.
- Patient ID is f10f8d972aba4fd^^^&188.8.131.52.4.1.21367.2005.13.20.3000&ISO
- Dataset contains the following submissions
- Submission 1 (single DocumentEntry)
- SubmissionSet (title=Physical) (id=urn:uuid:0601a770-f7d6-4592-9323-ab2cb693da3e) (uniqueId=1.42.20141105202336.38)
- DocumentEntry (title=DocA) (id=urn:uuid:d5bc063e-b960-4d55-b05d-97ec850a6ef0)(uniqueId=1.42.20141105202336.37)
- Submission 2 (one DocumentEntry, one Folder, DocumentEntry placed in Folder)
- SubmissionSet (title=Physical) (id=urn:uuid:f3df9f3c-b129-4bea-ada7-d15cd2a51c51) (uniqueId=1.42.20141105202336.40)
- Folder (title=FOLDER) (id=urn:uuid:168aafdf-c1e1-4b49-bdd2-97a6500ed575) (uniqueId=1.42.20141105202336.41)
- DocumentEntry (title=DocB) (id=urn:uuid:0728d511-c6a1-420b-83d0-8ae8f4ee35c3) (uniqueId=1.42.20141105202336.39)
- Submission 3 (two DocumentEntries, one Folder, both DocumentEntries placed in Folder)
- SubmissionSet (title=Physical) (id=urn:uuid:f535a28e-116e-4a20-b5dc-92a1980c5c98) (uniqueId=1.42.20141105202336.44)
- Folder (title=FOLDER) (id=urn:uuid:76583963-c939-40d0-b11c-9ed21fb35f3a) (uniqueId=1.42.20141105202336.45)
- DocumentEntry (title=DocC) (id=urn:uuid:22adcc04-cd74-4d03-857e-8b6b236c7f24) (uniqueId=1.42.20141105202336.42)
- DocumentEntry (title=DocD) (id=urn:uuid:ad8706dc-d0d6-4df9-8c9e-af697d1f6067) (uniqueId=1.42.20141105202336.43)
- Submission 4 (single DocumentEntry - preparing for next submission where it is replaced)
- SubmissionSet (title=Physical) (id=urn:uuid:02f6b496-1f2b-4182-af32-aac038fbbe98) (uniqueId=1.42.20141105202336.47)
- DocumentEntry (title=DocE) (id=urn:uuid:c94a4319-75ec-46b9-91af-8c396eb64891) (uniqueId=1.42.20141105202336.46)
- Submission 5 (single DocumentEntry replaces DocE)
- SubmissionSet (title=Physical) (id=urn:uuid:d3fb7021-c587-42ca-93c9-8421f59fca54) (uniqueId=1.42.20141105202336.49)
- DocumentEntry (title=DocF) (id=urn:uuid:7f1413dc-ca71-4cc3-94a1-8fca23d264bf) (uniqueId=1.42.20141105202336.48)
- Submission 1 (single DocumentEntry)
Note: I have unregistered these Patient IDs to prevent others from adding to these collections.
To examine this data
- In your favorite browser bring up the XDS Toolkit at http://ihexds.nist.gov:12080/xdstools2
- Select the Find Documents tool
- Copy/paste the Patient ID into the Patient ID box
- Disable TLS
- Select Document Registry - pub
- Press the 'Run' button
- Press the 'Inspect Results' button (which will let you browse data and issue additional queries)