Reading Audit Events with rwslib

written by Ian Sparks
23 Dec 2014
python, Rave, EDC, RWS, webservices, audit

In our last post we introduced rwslib, a Python client library for Rave Web Services (RWS). In this post we look a little closer at one of the components of RWS: the Clinical Audit Records web service and at a rwslib helper that makes it easy to capture key events from the Rave EDC System.

RWS provides several webservice endpoints to extract clinical data in CDISC ODM and CSV formats. This data is ideal for reporting since it represents the latest available values entered into the system. But what if you are trying to populate a data warehouse and want to be able to show point-in-time values or look at the pattern of changes to data over time?

In that case you want access to the audit trail which logs every change made to the data through its lifetime. Happily, RWS provides access to a large set of that data through its Clinical Audit Records service. The Clinical Audit Records (audits) service allows you to treat the audit trail as a kind of tape, reading from the first entry up to the present time. Typically the audit trail is a very large table of data so the output of the service is batched, each time you make a request of the service it includes a header in the response to let you know if there is more data to be read and what URL should be called to retrieve the next time-ordered batch. Many types of audits can be extracted from the system.

What does an example response look like? Here’s a snippet:

    <?xml version="1.0" encoding="UTF-8"?>
    <ODM xmlns="http://www.cdisc.org/ns/odm/v1.3" xmlns:mdsol="http://www.mdsol.com/ns/odm/metadata" ODMVersion="1.3" 
         FileType="Transactional" FileOID="4d690eda-4f08-48d1-af26-3bab40f6118f" CreationDateTime="2014-11-04T16:37:05">
      <ClinicalData StudyOID="EXAMPLE(DEV)" MetaDataVersionOID="2867" mdsol:AuditSubCategoryName="SubjectCreated">
        <SubjectData SubjectKey="538bdc4d-78b7-4ff9-a59c-3d13c8d8380b" mdsol:SubjectKeyType="SubjectUUID" 
                     mdsol:SubjectName="01" TransactionType="Upsert">
          <AuditRecord>
            <UserRef UserOID="isparks" />
            <LocationRef LocationOID="1001" />
            <DateTimeStamp>2014-08-13T10:40:06</DateTimeStamp>
            <ReasonForChange />
            <SourceID>6434193</SourceID>
          </AuditRecord>
          <SiteRef LocationOID="1001" />
        </SubjectData>
      </ClinicalData>
      <ClinicalData StudyOID="EXAMPLE(DEV)" MetaDataVersionOID="2867" mdsol:AuditSubCategoryName="Entered">
        <SubjectData SubjectKey="538bdc4d-78b7-4ff9-a59c-3d13c8d8380b" mdsol:SubjectKeyType="SubjectUUID" 
                     mdsol:SubjectName="01">
          <SiteRef LocationOID="1001" />
          <StudyEventData StudyEventOID="SUBJECT">
            <FormData FormOID="EN" FormRepeatKey="1">
              <ItemGroupData ItemGroupOID="EN">
                <ItemData ItemOID="EN.USUBJID" TransactionType="Upsert" Value="01">
                  <AuditRecord>
                    <UserRef UserOID="isparks" />
                    <LocationRef LocationOID="1001" />
                    <DateTimeStamp>2014-08-13T10:40:06</DateTimeStamp>
                    <ReasonForChange />
                    <SourceID>6434194</SourceID>
                  </AuditRecord>
                </ItemData>
              </ItemGroupData>
            </FormData>
          </StudyEventData>
        </SubjectData>
      </ClinicalData>
      <!-- 998 more ClinicalData Entries -->
    </ODM>

As you can see, it’s a CDISC ODM formatted XML document with one ClinicalData entry per audit trail entry. The contents of each ClinicalData element varies based on the type of audit entry. In our example above the first entry is related to the creation of a new subject record so there is no StudyEvent, Form or Item level data elements. The second entry relates to a change of a form fields’ value so it has this extra context information to allow the precise identification of its name and location.

Audits and rwslib

rwslib provides a helper for the audits service called AuditRecordsRequest which makes it easy to get access to this service:

>>> from rwslib import RWSConnection
>>> from rwslib.rws_requests.odm_adapter import AuditRecordsRequest
>>> rws = RWSConnection('innovate','my_username','my_password)
>>> rws.send_request(AuditRecordsRequest('Mediflex','DEV'))
<ODM ODMVersion="1.3" FileType="Transactional" FileOID="29537e84-aa4f-486b-b8d3-60d0fa115959" .....
>>> rws.last_result.headers
{'link': '<https://innovate.mdsol.com/RaveWebServices/datasets/ClinicalAuditRecords.odm?studyoid=Mediflex%28DEV%29&startid=3842&per_page=1000>; rel="next"',
 'date': 'Mon, 22 Dec 2014 14:20:47 GMT', 
 'content-type': 'text/xml'}

AuditRecordsRequest doesn’t do much more than format the request URL and make the request for you, it doesn’t parse the resulting ODM XML or extract the next link url from the returned headers. It’s a building block to help you build your own integrations. But we can go a step further and make use of an additional library from the extras package in rwslib which offers a simpler way consume the audit event service.

audit_event

audit_event builds on the facilities of rwslib and AuditRecordsRequest to provide an event-style interface to the AuditRecordsRequest and the ODM data it returns. ODMAdapter is a class that manages an RWSConnection to request and parse the ODM and request headers for you. All you have to do is define handlers for the audit events that you want to capture and set it running.

Let’s imagine you want to capture a list of all subjects created from the audit trail. First you need to create a class which will respond to the SubjectCreated event.

class Subjects(object):
    def __init__(self):
        self.subjects = []

    def SubjectCreated(self, context):
        """Capture the SubjectCreated event"""
        self.subjects.append(context)

We will pass an instance of the above class to ODMAdapter and each time an ODM ClinicalData element with the SubjectCreated audit subcategory is processed the SubjectCreated method will be called with the data from the ODM passed in the context parameter (read about the structure of the context object). At the end of the run through the audits the subjects list will be populated with a list of context objects, one for each subject created.

Let’s see how that works:

    from rwslib import RWSConnection
    from rwslib.extras.audit_event import ODMAdapter

    #Make our RWS connection
    rws = RWSConnection('innovate','my_username','my_password)

    #Make our event handler
    subjects = Subjects()
    
    #Create ODMAdapter passing in the name of the study/environment we want to process and the event handler
    o = ODMAdapter(rws, "TESTSTUDY","DEMO", subjects) 
    o.run(start_id=1,per_page=1000)
    
    #When complete, subjects.subjects will contain the list of subjects found.

The ODMAdapter class has a number of options which can be set to control the size of the ODM document retrieved (how many entries it includes) as well as where in the audit trail to start.

All you need to make use of audit_event is Python, a copy of rwslib from GitHub (this includes the audit_event in the extras module) and a Rave account. The ClinicalAuditRecords service will only return audits that you have permission to view.

Interested in building reporting or other data extraction solutions using RWS and/or rwslib? We have a developer program. Please get in touch!