Xdstest2 testing tool
Starting with the 2007-2008 testing season, a new tool will be used to test XDS Registry and Repository actors. Tests run using the old tool, xdstest, will no longer be accepted for validation.
This new tool performs roughly the same function as the old tool, generated a transaction to an actor implementation under test based on a configuration file. Then accept the reply and save both the initiating transaction and the reply to a log file. The new tool has the following new features:
- Configuration and log files are both in XML format
- The configuration file can describe multiple test steps
- Patient ID and unique IDs are managed centrally and allocated automatically and are logged to the log file
- Knows about both XDS.a and XDS.b transactions
- The metadata in outgoing transactions and their responses are validated against both Schema and the XDS Metadata Validator before they are sent
- Validates response against test expectations (metadata structure etc.)
- Can contain multiple test steps. This is used in Registry testing, for example, to submit metadata and then query to validate the Registry actor's response.
Download the latest version of the package from here.
View change log here. <<=== Needs updating
Java installation: Java version 1.5 is required. This tool is written and compiled in Java version 1.5. It uses features not available in Java 1.4 and has not been tested with Java 1.6.
Installing the software: The tool is distributed as a single ZIP file named xdstest2tool.zip. Download and expand this file in a convenient location. The unzip will create a single directory xdstest2tool.
Configuring the tool: Located in xdstest2test installation directory is a Unix sh style script named xdstest2. If you are using a Unix-like system you can use this script to drive your tests. This script requires customization. A variable at the top must be updated to point to the location of the xdstest2test directory on your system.
Initializing the management area: to support the automatic allocation of IDs, xdstest2 uses a directory to store counters and configuration information regarding Patient IDs and UniqueIds. Inside the installation directory is a subdirectory named mgmt. It contains:
- uniqueid_base.txt - base of the OID for unique IDs
- uniqueid_index.txt - unique ID counter
Unique IDs are created by concatenating uniquid_base.txt and unique_index.txt. After each allocation, the value in unique_index.txt is incremented. As installed, the value in uniqueid_base.txt is not a valid base for OIDs. You must insert your own OID base. For testing, I use my IP address with a following period.
This directory also contains:
xdstest2 forms a Patient ID by concatenating patientid_base.txt, "^^^", assigning_authority.txt. As delivered it is configured with the correct Assigning Authority for the Public Registry. To use with tests that forward to the Public Registry (testing a Repository actor for example), allocate a patient ID as instructed in the test. The part before ^^^ should be stored in patientid_base.txt and you should ensure that the Assigning Authority part matches the contents of assigning_authority.txt. For tests that do not involve the Public Registry, use values of your choosing.
The file default.xml holds the homeCommunityId attribute for the Receiving Gateway being tested. XCA tests use this configured value to validate the return values from the RG. It needs to be correctly configured.
Actor configuration: the xdstest2tool directory contains a file actors.xml. This can be used to control which actor is being tested. This is a new feature and not yet supported.
Testing the installation
After completing the installation, I suggest running test 11710 to verify the correct operation.
To run a test against a Registry or Repository actor implementation, start a shell window (or equivalent) and change directory to the directory defining the test to be run. For example, to run test 11827 (Repository accepts Provide and Register containing a single document via the XDS.a specification):
> cd testkit/tests/11827 > ls my_document.txt single_doc.xml testplan.xml > # edit testplan.xml here > xdstest2 > xmlformat log.xml | less
Xdstest2 always works on the test defined in the current working directory. All tests have a minimum of one file: testplan.xml, which contains the test definition. When the test completes, the file log.xml will be generated. As shown above, testplan.xml must be edited before using it. As delivered, it points to the Public Repository as the unit under test. This editing is discussed in the next section. After running the test with xdstest2, I show the use of an open source program xmlformat which pretty-prints the xml and displays it. Obviously you should use the tools you are comfortable with.
Editing testplan.xml: the contents of this file for test 11827 are:
<?xml version="1.0" encoding="UTF-8"?> <TestPlan> <RegistryEndpoint>http://localhost:9080/axis2/services/xdsrepositorya</RegistryEndpoint> <TestStep id="submit_doc"> <ExpectedStatus>Success</ExpectedStatus> <ProvideAndRegisterTransaction> <XDSa/> <MetadataFile>single_doc.xml</MetadataFile> <Document id="Document01">my_document.txt</Document> </ProvideAndRegisterTransaction> </TestStep> </TestPlan>
The top element is always TestPlan which contains one or more TestStep elements. The RegistryEndpoint element is the Web Services Endpoint that the test message will be sent to. As shown, it points to the Public Registry. Edit this element to insert the endpoint of your implementation.
Some useful options to xdstest2 are:
- --prepare - prepare the output, save it to log.xml but don't send the transaction.
- xdstest2 stepname stepname ... - when a test plan includes multiple test steps, this form of the xdstest2 command allows you to execute only a single or a few of the test steps.
Understanding a TestPlan
A TestPlan is made up of multiple test steps, each named by the id attribute. The metadata produced by the test step is either coded inline inside a <Metadata/> element or referenced in a separate file via a <MetadataFile/> element. Test steps that generate ebRIM 3.0 format, also known as XDS.b format, are a little tricky to interpret. All metadata used as input to this tool is in ebRIM 2.1 format. If the test is to generate ebRIM 3.0 format the tool translates as needed. The instruction that causes the translation to ebRIM 3.0 format is <XDSb/>. It will be found inside the <TestStep/> element. To see the exact metadata generated, inspect the <InputMetadata/> element of the log.xml file. Each test step is represented in log.xml so look inside the correct test step.
All test plans come configured to run against the Public Registry. That is how they are tested. To see the actual request/response generated by the test plan/Public Registry combination just run the test plan as delivered then inspect the <InputMetadata/> element to see what was sent and inspect the <Result/> element to see what was received back.
Interpreting log.xml files
Each test, upon completion, generates an log.xml file containing the result of the test. The log file for 11827 looks like:
<TestResults status="Pass"> <RegistryEndpoint>http://localhost:9080/axis2/services/xdsrepositorya</RegistryEndpoint> <TestStep id="submit_doc" status="Pass"> <ExpectedStatus>Success</ExpectedStatus> <ProvideAndRegisterTransaction> <MetadataFile>./single_doc.xml</MetadataFile> <RegistryEndpoint>http://localhost:9080/axis2/services/xdsrepositorya</RegistryEndpoint> <AssignedPatientId> <Assign symbol="SELF-5^^^&220.127.116.11.4.1.21367.2005.3.7&ISO" id="ST325^^^&18.104.22.168.4.1.21367.2005.3.7&ISO" /> </AssignedPatientId> <AssignedUids> <Assign symbol="SubmissionSet01" id="22.214.171.124.1709" /> <Assign symbol="Document01" id="126.96.36.199.1708" /> </AssignedUids> <InputMetadata> <rs:SubmitObjectsRequest xmlns:rs="urn:oasis:names:tc:ebxml-regrep:registry:xsd:2.1"> <rim:LeafRegistryObjectList xmlns:rim="urn:oasis:names:tc:ebxml-regrep:rim:xsd:2.1"> <rim:ObjectRef id="urn:uuid:7edca82f-054d-47f2-a032-9b2a5b518fff" /> <rim:ObjectRef id="urn:uuid:a54d6aa5-d40d-43f9-88c5-b4633d873bdd" /> <!-- lots of XML omitted here --> </rim:LeafRegistryObjectList> </rs:SubmitObjectsRequest> </InputMetadata> <Result> <rs:RegistryResponse xmlns:tns="http://serviceclasses.ws.registry.nist.gov" xmlns:rs="urn:oasis:names:tc:ebxml-regrep:registry:xsd:2.1" status="Success"> </rs:RegistryResponse> </Result> <StepStatus>Success</StepStatus> </ProvideAndRegisterTransaction> </TestStep> </TestResults>
There are several important things to look for. The top element, TestResults has a status attribute that tells whether the test passed or failed. Each TestStep also contains a status attribute. The InputMetadata and Result element contain what was sent to the actor under test and the response. Under TestStep there may be an Error element holding an error message if the response was not as expected. TestResults may have a FatalError element if an internal error is detected. The ExpectedStatus element indicates the test expected a successful response. Some tests require the actor under test to return errors.
It is important to understand what gets sent to the unit under test. Note in the previous example the element <XDSa/>. This signals the tool to submit the XDS.a version of the metadata. The alternate value is <XDSb/> which causes the XDS.b version of the metadata to be sent. As delivered, each test plan includes XDS.a metadata. Tests that include the <XDSb/> element internally convert the metadata to XDS.b format. To understand exactly what was sent to your implementation, inspect the following elements of the teststep structure of log.xml:
- <InputMetadata/> contains the exact message content transmitted. It is transmitted as the child of the SOAP:Body element. Note that for XDS.b Provide and Register transactions, this includes the encoded form of the attached documents.
- <OutHeader/> contains the contents of the SOAP:Header that was sent.
The reply message is documented as follows in log.xml:
- <Result/> contains the SOAP:Body of the reply message
- <InHeader/> contains the SOAP:Header of the reply message
The AssignedPatientId element shows the patient ID that was inserted into the metadata before submitting. The AssignedUids element shows the unique IDs that were allocated and sent. The id attribute shows which metadata object received a unique ID. The id attribute, as shown is taken directly from the id attribute of either the ExtrinsicObject or RegistryPackage element in metadata.
There are two kinds of query tests in the testkit. First, any time a Register transaction is being tested, a query must be executed to validate the contents of the Registry. This usually uses a GetSubmissionSetandContents Stored Query. Other tests focus on detail properties of a particular Stored Query so a corpus of test data must be submitted to the Registry to support the queries. In either case it is necessary to know either a patient ID or unique ID to perform the query. xdstest2 offers linkage between submission and query so that patient ID and unique ID do not need to be hand coded into the query. Looking at part of a log.xml file from a submission:
<TestStep id="basic"> <Rule> <Transaction>SQ</Transaction> <SQ>FindDocuments</SQ> <Assertion>Basic operation with patient ID and status</Assertion> </Rule> <ExpectedStatus>Success</ExpectedStatus> <StoredQueryTransaction> <Metadata> <query:AdhocQueryRequest xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:query="urn:oasis:names:tc:ebxml-regrep:xsd:query:3.0" xmlns="urn:oasis:names:tc:ebxml-regrep:xsd:rim:3.0" xmlns:rs="urn:oasis:names:tc:ebxml-regrep:xsd:rs:3.0"> <query:ResponseOption returnComposedObjects="true" returnType="LeafClass"/> <AdhocQuery id="urn:uuid:14d4debf-8f97-4251-9a74-a90016b0af0d"> <Slot name="$XDSDocumentEntryPatientId"> <ValueList> <Value>'$patient_id$'</Value> </ValueList> </Slot> <Slot name="$XDSDocumentEntryStatus"> <ValueList> <Value>('urn:oasis:names:tc:ebxml-regrep:StatusType:Approved')</Value> </ValueList> </Slot> </AdhocQuery> </query:AdhocQueryRequest> </Metadata> <UseId testdir="../submit1" id="SubmissionSet01" step="submit_doc" section="AssignedPatientId" symbol="$patient_id$"/> <ExpectedContents> <Documents count="4"/> </ExpectedContents> </StoredQueryTransaction> </TestStep>
Note the UseId element. This makes reference to another submission and directs the tool to pull information out of that log.xml file.
The relevant part of the log.xml from the referenced submission is:
<RegisterTransaction> <MetadataFile>/Users/bill/IheOs/workspace_prod/xds/selftest/submit1/single_doc.xml</MetadataFile> <AssignedPatientId> <Assign symbol="patient_id" id="ST68^^^&188.8.131.52.4.1.21367.2005.3.7&ISO"/> </AssignedPatientId> <AssignedUids> <Assign symbol="SubmissionSet01" id="184.108.40.206.1053"/> <Assign symbol="Document01" id="220.127.116.11.1052"/> </AssignedUids> <AssignedUuids> <Assign symbol="SubmissionSet01" id="urn:uuid:7ca8e601-4149-4d0c-8e52-b5afeaa7a771"/> <Assign symbol="Document01" id="urn:uuid:88201175-8b15-4118-9558-900c443a2d60"/> </AssignedUuids> ...
The UseId element in the query test step is:
<UseId testdir="../submit1" id="SubmissionSet01" step="submit_doc" section="AssignedPatientId" symbol="$patient_id$"/>
and can be interpreted as:
- testdir - directory holding the log.xml
- step - focus on the TestStep with id="submit_doc"
- section - within that TestStep, look inside the AssignedPatientId section. Submissions only submit to a single patient ID so the id attribute is ignored.
- symbol - is the pattern in the query specification (see the QueryTransaction/Metadata section) that is to be replaced with the patient ID extracted from the referenced log.xml
UseId is also used to reference a submission element by unique ID. An example is
<UseId testdir="../submit1" id="SubmissionSet01" step="submit_doc" section="AssignedUids" symbol="$ssuid$"/>
which can be interpreted as:
- testdir - directory holding the log.xml
- step - focus on the TestStep with id="submit_doc"
- section - within that TestStep, look inside the AssignedUids section
- id - within the AssignedUids section, look for symbol="SubmissionSet01"
- symbol - replace every instance of symbol ($ssuid$) in the query with the value SubmissionSet01
The UUIDs can be extracted from log.xml files in the same way. The section name is different (AssignedUuids).
If you are ever confused regarding what the tool actually sent, look in the InputMetadata element. This shows the metadata as it was submitted.
Why do XDS.b tests contain XDS.a metadata in testplan.xml and the files it references?
The xdstest2 tool converts to the version of metadata needed for the test. If you notice the command <XDSb/> inside a <ProvideAndRegisterDocumentSet/> element, that tells the tool to convert to XDS.b format before sending. To see the exact metadata that was send, look in the <InputMetadata/> section of log.xml.