XDS Test Management
From IHEWiki
Contents |
Generic Testing Procedures
Generic Testing Procedures Using xdstest
- Install and configure xdstoolkit (which includes xdstest tool)
- Install and configure testkit (which includes individual test descriptions and scripts)
- Review general xdstest instructions here
- Review general discussion of test from here. This page holds a description of each test indexed by test number.
- Review detailed instructions for test. They are found in testkit/tests/TESTNUMER/readme.txt
- Run each part using guidance from here
- Verify success using guidance from here
- Report the results using guidance from here
Generic Testing Procedures Using Public Registry Server
- Review configuration information for the Public Registry server found here.
- Review instructions for test from here. This page holds a description of each test indexed by test number.
- Obtain a Patient ID and Unique IDs as necessary
- Run the test steps
- Report the results using guidance from here
Public Registry Testing Services
The following services are available on the Public Registry:
- Document Repository. This is a full implementation of both the XDS.a and XDS.b Document Repository actor.
- Document Registry. This is a full implementation of both the XDS.a and XDS.b Document Registry actor.
- Test Log facility. This captures all activity sent to the above two services. The next section describes how this facility is used in testing.
- Audit Repository. This is a full implementation of the BSD Syslog Collector.
- Audit Log Browser. This is used to browser the contents of the Audit Log Collector.
- Tools page. This allows a manual way to allocate a Patient ID for the Public Registry (this may be obsolete now that the xdstest tool does it for you). A tool is include for validating metadata against Schema and the Metadata Validator (validates XDS specific rules as well as rules from the base standard that are beyond the capabilities of XML Schema).
Overall Structure
Tests for XDS and related profiles come in two forms and are supported by two testing tools. For testing your server (IHE actor implementation accepts connections from a client), a test client called xdstest is used. For testing your client (IHE actor implementation generates connections to a server) a server called the Public Registry is used.
Server Testing
Server Testing Background
For server testing with xdstest, there are two necessary parts. The xdstest tool is a command line tool written in Java. Supporting this tool is a collection of test data referred to as the testkit. The testkit generally contains individual tests named by a test number (11765 as an example). These test numbers correlate with the test numbers in Kudu, the IHE test management tool which assigns tests and accepts test results. In the testkit each test number is a directory. In its simplest form, each test directory contains a testplan.xml file and some supporting files. The testplan.xml is a test script and the xdstest tool is the script interpreter. Instructions on how to use xdstest can be found here. More complicated tests contain sub-directories named 'submit', 'eval' etc. These represent individual steps you are likely to want to run separately. Each sub-directory contains a testplan.xml file possibly with other supporing files. Each test directory contains a readme.txt file which describes the test and its individual steps. Another file, index.idx, is a text file that lists the order the sub-directory sub-tests are to be run. The xdstest tool can be run each individual test step separately or can run the entire test at once. It uses the index.idx file to know what order to run the sub-tests.
Each testplan.xml file contains one ore more <TestStep/> elements which are the atomic actions to be performed. Again like the sub-tests, the test actions can be run all together or one at a time.
The execution of a testplan.xml file results in a log.xml file. Whether you run one test step or all test steps, a single log.xml file is generated.
The xdstest tool is part of a growing collection of command lines tools referred to as the xdstoolkit. When you download xdstest you are actually downloading the entire xdstoolkit.
Tests are now managed in collections based on the transaction being tested. See the testkit/collections directory. A new option on xdstest (-tc) can be used to point the tool to a collection of tests to be run.
Unlike the operation of xdstest in prior years, there is no longer any need to edit testplan.xml files. This is because the following information is configured into the tool and its support files:
- The Patient ID for testing
- Unique ID generation (as before)
- Web Service Endpoint (new this year) now comes from the configuration
Patient IDs are automatically generated as needed with a Web Service call to the Public Registry. This means that the entire suite of tests for a particular actor can be run automatically.
Evaluating the test
The log.xml files generated can be inspected for debugging information as needed. Each test step results in a section of the file documenting is operation. Each test step is labeled with an attribute, status, indicating the pass/fail status of the test. The top most element also has a status attribute indicating the results of the entire test event. If the top most element has status=Failure then one or more test steps has status=Failure.
All XDS metadata included in the testkit is in ebRIM 2.1 format (XDS.a). The tool converts this to ebRIM 3.0 (XDS.b) format as needed. To see the actual metadata sent, look inside log.xml at the test step of interest. Under the test step will be an element with a name ending in Transaction (RegistryTransaction is an example). Inside that element is an element InputMetadata. This contains the actual metadata sent in the transaction. Other sections inside the *Transaction section include AssignedPatientId, AssignedUids, AssignedUuids (some tests assign UUIDs while other let the Registry actor do it), OutHeader (output SOAP header), InHeader (returned SOAP header), and Result (the ebRS and ebRIM content of the response message).
When requesting assistance with testing issues, please include the log.xml file in your correspondence.
Client Testing
To test your client, Document Source or Document Consumer for example, the Public Registry is used. This is an Internet server that is available 12 months a year and supports many testing events. The URLs/Endpoints used are specialized for each testing event. The server configuration is documented here.
Your client, Document Source or Document Consumer for example, must manage Patient IDs and UniqueIds as a normal part of its operation. A Document Source implementation must have access to a Patient Id management module (probably with access to an ADT system). When testing against the Public Registry, a web based Patient ID allocator here must be used. This not only allocates a Patient Id but also registers it so the Document Registry implementation on the Public Registry server will accept it.
Most tests can be run using a Patient ID that you have used for other tests. If new 'empty' Patient ID is needed for a test then the test documentation will indicate so.
For the normal operation of your Document Source actor implementation, it must allocate UniqueIds for many XDS metadata objects. No tooling is available to assist with this but the Document Registry implementation on the Public Registry does check that these ids are truly unique.
Configure Public Registry Endpoint
When a client sends to the Public Registry server, the correct WebService Endpoint must be used. Endpoints are organized by events. Each event uses different endpoints. This allows different configurations, like authentication certificates, to be established for each event. In the test documentation an endpoint is documented as
http://ihexds.nist.gov:PORT/EVENT/services/test11730
There are several things to note about this endpoint:
- PORT is a placeholder for the actual port number
- EVENT is a placeholder for the actual event code
- test11730 is a specialty endpoint for validating the results of this test. Both specialty endpoints and general purpose endpoints are offered. The general purpose ones operate per the profile specification. The specialty versions add test specific validations.
Look here to find the values for PORT and EVENT to use.
Server and Client Testing
Some actors act as both client and server. The XDS Document Repository actor fits this description. It accepts messages from a Document Source actor and sends messages to a Document Registry actor. For testing Document Repositories, xdstest is used to generate the message to the Document Repository. The Document Repository is configured to send to the Document Registry available on the Public Registry server. This way both sides of the Document Repository under test can be monitored.
Using xdstest
While the most detailed (and most up to date) documentation on xdstest is gotten by executing xdstest:
xdstest -help
these notes will help you get started.
Basically, xdstest operates on a testplan.xml file (a script file for xdstest) in the current copy of the testkit. The most frequent invocation is:
xdstest -run -err -t 11890
This call does the following:
- Finds the test 11890 in the testkit (-t 11890). The testkit is found via an environment variable established when the testkit is installed.
- Finds the log directory to deposit the log.xml file. This directory is found via environment variable established when the testkit is installed. The log directory follows the directory structure of the testkit.
- Runs the test (-run). The log.xml file is deposited into the log directory.
- At the end of the run, an error summary is displayed (-err).
Also in the background the following things happened:
- The current Patient ID was pulled from the configuration area.
- If the test plan called for a new Patient ID to be allocated then the Public Registry was called (via a Web Service) to allocate a new Patient ID. Since the allocation came from the Public Registry if this test is for a Document Repository that sends metadata to the Document Registry actor on the Public Registry the ADT Patient ID registration is already handled.
- New Unique IDs were allocated as needed.
- Endpoints for the underlying Web Service were taken from the configuration. The above invocation did not specify a site (my term for machine hosting a bunch of services). The default site was used. The file actors.xml in the configuration area was referenced. Documentation inside that file describes how to add new sites (like the actors you want to test), how to set the default site, and the WS Endpoints for each transaction on that site. It also includes the pointer to the Public Registry for Patient ID allocation. Each site has its own allocator but you are welcome to use the Public Registry server.
Basically, once the testkit is installed and xdstest is configured, you can forget about the testkit. Well, that is not quite true. Each test directory contains a readme.txt file that gives detailed directions for running the test. So, you can almost forget where you installed it.
How to report results
Tests run with xdstest tool
These tests are described by a test description file testplan.xml and generate a results file log.xml. Generally these are tests for Document Repository and Document Registry actors. These tests have one of two structures:
- A single testplan.xml file which generates a single log.xml file. To document your test completing, upload the log.xml file to Kudu. Each test has a button to 'upload a log file'.
- Some tests have multiple directories each containing a testplan.xml. These tests will generate multiple log.xml files. To report the test results, zip the log.xml files together and upload. The log.xml files can be renamed before zipping.
It is important to follow these directions. Test validation is done by a background script. Your test results will be ignored an not graded if these instructions are not followed.
Tests run against the Public Registry
These tests are for Document Source, Document Consumer, or Initiating Gateway actors. Have your actor under test interact with the Public Registry server which implements the Document Repository, Document Registry, and Responding Gateway actors. Each interaction with the Public Registry server generates entries in the Test Log Service. You can browse your entries in the Test Log Service using the special browser found at http://ihexds.nist.gov/LogReader. This browser automatically senses your IP address and only shows test entries generated from your machine. Your IP address is displayed at the top (center) of the window. Users typically have two problems using this browser:
- The test message is generated from one machine and they wish to browse from another machine. This can be done by selecting the IP address in the window and typing in the correct IP address.
- Some corporate firewalls map IP addresses so that the IP address you see on your machine is different from the IP address reported when you connect to services, like the Public Registry, outside your firewall. If you have problems with this you need to contact your IT department and find out what IP address they generate for you. You can then enter this public IP address into the Test Log Browser.
Once you have the browser working, you can find the results of your testing. To report test results, select the test event generated by your work. At the top right corner of the window you will see a link 'Download EVS'. Select this link and your browser will download a small (10-12 lines) XML file that contains the necessary links to your entries in the test log. Take this XML file and upload it to Kudu as a log file. If done correctly your test will be validated automatically.
Note that the automatic validation does not happen instantly. It may take several minutes or hours for the server to find your results and process them.
Syslog tests
The Public Registry server hosts a syslog service that you can send your audit message to. For many XDS related ATNA tests you are required to send to this audit repository. A browser exists, similar to the Test Log Browser, for finding your audit records. Later in the season a 'Download EVS' link will be added to the browser to allow test result reporting, just like the above use of the Test Log Browser.
Evaluations
The following evaluations are done on metadata when:
- Received by the Public Registry server
- Generated by xdstest (before sending)
- Received by xdstest as a response
Schema
Metadata validated against appropriate Schema(s) as published by Oasis and/or IHE.
Metadata Validator
Tool that checks for metadata structure required by the standard but beyond the capability of XML Schema. Also checks for structural and coding rules imposed by the XDS profile.
Test Data
Some minimal test data is kept in the Public Registry server Document Repository and Document Registry actors.
- Generated by testkit testkit/testdata/11890. This data does not have real URI or documentUniqueId values. It is only good for testing Stored Query. The Patient ID is 66b5b2f2a6454de^^^&1.3.6.1.4.1.21367.2005.3.7&ISO
- Generated by testkit testkit/testdata/12344. This data has good URI and documentUniqueId values and real documents in the Repository (for now all simple text documents). The Patient ID is d0a6ca0913fb456^^^&1.3.6.1.4.1.21367.2005.3.7&ISO
- Generated by testkit testkit/testdata/12361. There are two Patient IDs: fb569b6637d340c^^^&1.3.6.1.4.1.21367.2005.3.7&ISO (with a single document) and ab31f8cd00b834ad^^^&1.3.6.1.4.1.21367.2005.3.7&ISO (with two documents).
To examine this data
- In your favorite browser bring up the XDS Viewer at http://ihexds.nist.gov/xdstools/viewer
- Select pub (public registry)
- Copy/paste the above Patient ID into the id: box.
- Press the 'Doc' button.
The available documents will be listed in the left panel. Select one to see the metadata details on the right.
Tests that generate test data
The following tests generate the same test data but in different environments:
Registry metadata only (document references are fake)
11890 - Uses Register.a transaction to load Document Registry
12346 - Uses Register.b transaction to load Document Registry
Installs real documents in Repository
12328 - Uses Provide and Register.a to load Document Repository and Document Registry
12345 - Uses Provide and Register.b to load Document Repository and Document Registry
