1 Oracle Enterprise Data Quality Applications Help Topics

1.1.4.1 Adjusting the Client Locale

A client machine will display the EDQ UIs in the local language according to the machine's display settings.

To adjust the language of the client in order to change the language of the UIs, use the following procedure:

1. Set the web browser's language display options to display web pages from the EDQ server in the chosen language.

2. Change the regional settings of the client machine in order to display the EDQ Java WebStart UIs in the chosen language; for example, on Windows machines change the System Locale, Format, and Display Language.

   Notes:

   • Following the Java 7 Update 25, the Display Language now must be adjusted in order to modify the language in which Java applications are displayed. In previous versions, only the System Locale and Format were used. If you are using Windows, only Windows Enterprise and Windows Ultimate include the Multilingual User Interface Pack required to alter the Display Language from its installed setting.

   • For testing purposes, it is possible to override the client settings using a server option that sets the locale for all clients. To do this, add the following setting to [edq_local_home]/properties/clientstartup.properties: locale = [ISO 639-2 Language Code]. For example, to make all client Java UIs appear in Japanese regardless of the client's regional settings, add this line: locale = ja

1.2.1.1 Getting Started with Director

To get started with Director, see "Getting Started" in Oracle Fusion Middleware Using Oracle Enterprise Data Quality.

1.2.1.2 Menu

The Menu contains four submenus, as detailed below:

• File Menu

• Edit Menu

• View Menu

• Help Menu

1.2.1.3 Toolbar

The toolbar provides easy access to a number of common functions in EDQ. The icons on the toolbar represent these common functions.

Tooltips are available so that when you hover over the icon on the toolbar, a brief description of its function is displayed.

1.2.1.4 Issue Notifier

The Issue Notifier shows how many open issues are currently assigned to you on the server to which you are connected.

Click on the Issue Notifier to launch the Issue Manager and display the details of the issues assigned to you.

1.2.1.5 Project Browser

The Project Browser allows you to browse an EDQ server, and the projects contained in it.

• Projects

• Reference Data

• Data Stores

• Published Processors

Note that you can copy most items in the Project Browser to new projects, or other EDQ servers, by dragging and dropping the item, or by using Copy (Ctrl+C) and Paste (Ctrl+V).

Right-click on a blank area of the Project Browser to connect to a new EDQ server, or to open a package file.

Project Browser Object States

Objects in the Project Browser may appear with different overlay icons depending on their state. A green 'Play' icon indicates the object is running, an amber 'Stop' icon indicates that execution of the object was canceled, and a red 'Warning' icon indicates that execution of the object resulted in an error.

These states are:

• Normal

• Running

• Canceled

• Errored

There are two types of lock. A downwards-pointing red triangle indicates that the object is locked and cannot be opened (most likely because it is open for editing by another user). A blue dot indicates that the object is read-only (most likely because it is currently being used in a job and cannot be edited in order to preserve the logic of the job).

The lock states are:

• Locked (the object cannot be viewed)

• Read-Only (the object can be viewed, but not edited)

1.2.1.6 Canvas

The Canvas is where processes are opened, and therefore where data quality processes are designed using EDQ.

Many processes may be open on the Canvas at any one time.

You may close an individual process by selecting Right-click, Close, or by clicking on the Close process icon in the top-right of the canvas:The Canvas has its own toolbar for Canvas-specific actions.

Note that most of the main Toolbar functions can also be used for the Canvas.

Note that the process on the canvas may appear differently depending on its state. See Process states.

The processors in a process also change their appearance depending on their state. See Processor states.

If a process is open on the canvas when it is executed, you can view the progress of each processor in the process, and monitor the overall progress of the process. Otherwise, you can monitor its progress in the Task Window.

1.2.1.7 Tool Palette

The Tool Palette provides a list of all the available processors that you may use when working on projects in EDQ.

The processors are listed by their Processor family.

To use a processor in the definition of a process, drag it from the Tool Palette and drop it onto an open process.

You can then connect a processor's inputs and configure it for use in your process.You can also find a processor by searching for it using the search box at the bottom of the tool palette. This will quickly search for processors using the entered text. For example, to find all processors with a name containing the word 'length', simply enter 'length' in the search box:

1.2.1.8 Canvas Overview Pane

The overview pane is used to aid navigation around large processes, which cannot be viewed in their entirety on the Canvas.

The overview pane displays a thumbnail view of the whole process, with a rectangular overlay that shows the area currently visible on the canvas.

You can move around the process rapidly by dragging the rectangle to the area you want to examine. The Canvas will automatically move to the framed area.

1.2.1.9 Task Window

The Task Window allows you to see the progress of all currently running tasks, including jobs, processes and snapshots, on all connected servers. Tasks are grouped by server, and overlays are used to show the state of the task.

Jobs in the task window can be expanded to show more detailed information about the phases and processes involved in that job.

Right-clicking on a task displays a context-sensitive menu containing options available to the selected task.

Running tasks can be canceled. Tasks which have errored can be opened or edited. Errors can be displayed, or cleared. Selecting the Clear All Errors option removes all errored tasks from the task window.

1.2.1.10 Results Browser

The Results Browser is designed to help you to understand your data by providing an interactive way of browsing the results of your processing using EDQ. Provided you are using the repository on process execution, you can drill down on the statistics in the Results Browser to useful views of the data that are designed to help you form business rules for validating and transforming it.

There are a number of general functions available in the Results Browser. These functions are accessible from the toolbar at the top of the Results Browser window.

1.2.1.11 System-level Reference Data Library

The following sets of System-level Reference Data are provided with EDQ.

Many of these lists and maps are used by processors by default. You may change these (by using them in projects, modifying them, and copying them back to the System library), though it is advisable to create new lists and maps with different names for your own needs, so that they will not be overwritten when EDQ is upgraded.

Oracle also provides packs of Reference Data for specific types of data, and for solving specific problems - for example, lists of known telephone number prefixes, name and address lists, and regular expressions for checking structured data such as URLs. These packs are available as extension packs for EDQ.

1.2.1.12 Run Profiles

Run Profiles are optional templates that specify a number of 'override' configuration settings for externalized options when a Job is run. They offer a convenient way of saving and reusing a number of configuration overrides, rather than specifying each override as a separate argument.

Run Profiles may be used when running jobs either from the Command Line Interface, using the runopsjob command, or in the Server Console UI.

Run Profiles can be created using any text editor. They must be saved with the .properties prefix to the oedq_local_home/runprofiles folder of the EDQ installation.

They are typically set up by an advanced user with knowledge of the ways configuration needs to be overridden in a production deployment. They may be created or edited directly by a user with access to the oedq_local_home directory, or transferred by an FTP task into the oedq_local_home/runprofiles directory. Solutions built using EDQ, such as Oracle Watchlist Screening, may include a number of pre-defined Run Profiles that are suitable for overriding externalized configuration options in pre-packaged jobs.

The template for creating Run Profiles is called template.properties, and can be found in the [Installpath]/oedq_local_home/run profiles directory. The template contains full instructions and examples for each type of override.

An example of a Run Profile file is included below.

######### Real-time Setup ###########
# Globally turns on/off real-time screening
phase.Start\ Real-time\ Screening.enabled = Y
# Control single real-time screening types
phase.Real-time\ Screening.process.Individual\ Real-time\ Screening.san_enabled = Y
phase.Real-time\ Screening.process.Individual\ Real-time\ Screening.pep_enabled = Y
phase.Real-time\ Screening.process.Individual\ Real-time\ Screening.edd_enabled = Y
phase.Real-time\ Screening.process.Entity\ Real-time\ Screening.san_enabled = Y
phase.Real-time\ Screening.process.Entity\ Real-time\ Screening.pep_enabled = Y
phase.Real-time\ Screening.process.Entity\ Real-time\ Screening.edd_enabled = Y
########## Batch Setup ##############
# Globally turns on/off batch screening
phase.Start\ Batch\ Screening.enabled = Y
# Control single batch screening types
phase.Match\ Individuals\ Batch\ SAN.enabled = Y
phase.Match\ Individuals\ Batch\ PEP.enabled = Y
phase.Match\ Individuals\ Batch\ EDD.enabled = Y
phase.Match\ Entities\ Batch\ SAN.enabled = Y
phase.Match\ Entities\ Batch\ PEP.enabled = Y
phase.Match\ Entities\ Batch\ EDD.enabled = Y
######## Screening Receipt ##########
phase.Real-time\ Screening.process.Individual\ Real-time\ Screening.receipt_prefix = OWS
phase.Real-time\ Screening.process.Individual\ Real-time\ Screening.receipt_suffix = IND
phase.Real-time\ Screening.process.Entity\ Real-time\ Screening.receipt_prefix = OWS
phase.Real-time\ Screening.process.Entity\ Real-time\ Screening.receipt_suffix = ENT

1.2.2.1 Managing Server Connections

The Server menu on Server Console is used to manage connections to servers.

On start-up, Server Console is connected to the server it was launched from. It is possible to connect to more than one server at a time, and these are separated into tabs in the Server Console window.

Connect to a Server

To connect to a server:

1. On the Server menu, click Connect.

2. Click Connect. The Login dialog is displayed.

3. Enter the login credentials, and click OK.

Disconnect from a Server

To disconnect from a server:

1. Ensure the correct server is currently displayed (select the relevant tab).

2. On the Server menu, click Disconnect.

3. Click OK on the Disconnect dialog.

Add a New Server

To add a new server:

1. On the Server menu, click New Server. The Add Server dialog is displayed.

2. Complete the fields, as described below:

   Table 1-5 New Server Fields

   Field Name	Format	Entry
   Alias	Free text	An alias to refer to the server in the Server Console UI.
   Host	Free text	The name of the server.
   Port	Numeric	The HTTP or HTTPS port to be used to connect to the server.
   Secure	Check box (unchecked by default).	Used to specify whether a secure connection should be established with the server.
   Path	Free text	The file path to the server.
   User	Free text	The username for connecting to the server.

   
   

3. Click OK to add the server.

Edit a Server

You can edit the details of a server that you have added.

To edit a server:

1. Select the required server.

2. Disconnect the server (see Disconnect from a Server above).

3. On the Server menu, click Edit Server.

4. On the Edit Server dialog, alter the details of the server as required. The fields are identical to those in the Add Server dialog (as described above).

5. Click OK to save changes.

Remove a Server

To remove a server:

1. Select the required server.

2. On the Server menu, click Remove Server.

3. On the Remove dialog, click OK.

1.2.2.2 Scheduler

The Scheduler window is used to run one-off instances of jobs and to create or edit job schedules.

The window is divided into three areas:

• Jobs - lists the jobs (by server, if more than one is available) that the user is authorized to run and schedule.

• Jobs Details - displays the details of the selected job or jobs.

• Schedules - lists the scheduled jobs.

Running a One-Off Job

To run a one-off job:

1. In the Jobs area, locate and double-click on the required job.

2. In the Job Details area, click the Run button. The Run dialog is displayed.

3. If available, select the required Run Profiles to override the settings for externalized configuration options in the Job.

4. Enter a Run Label to store the staged data results for the job against. You can either enter a new label, or select one from the drop-down list. Note: The drop-down list contains the last 100 used Run Labels.

5. Click OK to run the job.

Scheduling Jobs

The Schedule dialog is used to create and edit schedules for jobs:

Creating a new schedule

To create a new schedule:

1. In the Jobs area, locate and double-click on the required job.

2. In the Job Details area, click the Schedule button. The Schedule dialog is displayed.

3. Select the Schedule type and enter the date and time details (see Schedule Types below).

4. Select the Run Profiles if required.

5. Enter a new Run Label, or select one from the drop-down list.

6. Click OK to save.

Editing a schedule

To edit a schedule:

1. In the Schedules area, locate and double-click on the required job.

2. In the Job Details area, click the Schedule button. The Schedule dialog is displayed.

3. Select the Schedule type and enter the date and time details (see Schedule Types below).

4. Select the Run Profiles if required.

5. Enter a new Run Label, or select one from the drop-down list.

6. Click OK to save.

Deleting a schedule

To delete a schedule:

1. In the Schedule area, right click on the required schedule.

2. Select Delete.

3. In the Delete dialog, click Yes to delete or No to keep the schedule.

Schedule Types

There are five schedule types available:

Once

This option is for setting a job to run once only on a specified time and date.

1. Select Once on the Schedule dialog.

2. Enter the date and time required in the Run this job once on field. Either:

3. Change the day by clicking the up and down arrows on the right of the field:;

4. Change the day, month or year using the calendar on the right of the window; or

5. Edit the field manually, in the format shown: dd-MMM-yyyy hh:mm.

Daily

This option is for scheduling a job to run once a day, either every day or at intervals of a specified number of days (for example, every third day or every tenth day).

1. Select Daily on the Schedule dialog.

2. In the Every field, enter the schedule frequency; for example, 1 - the job runs every day, 3 - the job runs every third day, and so on.

3. In the Server Time field, enter the time of day at which the job will run, in 24hr format.

4. In the Start Date field, enter the date of the first day of the schedule. To do this, either select the required date from the calendar on the right of the screen, or manually edit the field in the format dd-MMM-yyyy.

Weekly

This option is for scheduling a job on a weekly basis.

1. Select Weekly on the Schedule dialog.

2. In the Every area, check each day of the week on which the job will run. You can select any number of combination of days.

3. In the Server Time field, select the time at which the job will run, in 24hr format.

4. In the Start Date field, enter the date of the first day of the schedule. To do this, either select the required date from the calendar on the right of the screen, or manually edit the field in the form dd-MMM-yyyy. Alternatively, to run the job on the first specified day of the week, clear the check box to disable the Start Date field.

Monthly

This option is for scheduling a job on a specific day of a month.

1. Select Monthly on the Schedule dialog.

2. In the On drop-down list, select the day of the month.

3. In the Server Time field, select the time at which the job will run, in 24hr format.

4. Weekends are included in the schedule by default; that is, if the date selected falls on a weekend, the job will run as scheduled. To exclude weekends, check the Exclude Weekends field.

5. In the Start Date field, enter the date of the first day of the schedule. To do this, either select the required date from the calendar on the right of the screen, or manually edit the field in the form dd-MMM-yyyy. Alternatively, to run the job on the first occurrence of the specified day of the month, clear the check box to disable the Start Date field.

Startup

This option sets the selected job to run whenever the server is started up.

1.2.2.3 Current Tasks

The Current Tasks window shows all tasks currently running on the selected server, including those initiated in the Director UI.

It is divided into two areas:

• Current Tasks area

• Task Filter

Current Tasks Area

This area displays the details of the tasks in progress. You can drill down into these tasks to check their status and progress by expanding each of the groups using the + button.

Task Filter

This area filters the details of the tasks currently running on the selected server.

The elements of this area are as follows:

Current Tasks Popup

This dialog box is used to view the tasks currently running on all connected servers.

To open the popup, click View > Current Tasks Popup.

Note that the Current Tasks window shows all activity on all connected servers, including tasks and jobs that are run interactively using the Director UI, and any jobs that have been instigated externally using the Command Line Interface.

1.2.2.4 Event Log

The Event Log provides a complete history of all jobs and tasks that have run on an EDQ server.

By default, the most recent completed events of all types are shown in the log. However, you can filter the events using a number of criteria to display the events that you want to see. It is also possible to tailor the Event Log by changing the columns that are displayed in the top-level view. Double-clicking on an event will display further information where it is available.

The displayed view of events by any column can be sorted as required. However, older events are not displayed by default, so a filter must be applied before sorting before they can be viewed.

Logged Events

An event is added to the Event Log whenever a Job, Task, or System Task either starts or finishes.

Tasks are run either as part of Jobs or individually instigated using the Director UI.

The following types of Task are logged:

• Process

• Snapshot

• Export

• Results Export

• External Task

• File Download

The following types of System Tasks are logged:

• OFB - a System Task meaning 'Optimize for Browse' - this optimizes written results for browsing in the Results Browser by indexing the data to enable sorting and filtering of the data. The 'OFB' task will normally run immediately after a Snapshot or Process task has run, but may also be manually instigated using the EDQ client by right-clicking on a set of Staged Data and selecting Enable Sort/Filter, or by a user attempting to sort or filter on a non-optimized column, and choosing to optimize it immediately.

• DASHBOARD - a System Task to publish results to the Dashboard. This runs immediately after a Process task has been run with the Publish to Dashboard option checked.

Server Selection

If the Director UI is connected to multiple servers, you can switch servers using the Server drop-down field in the top-left hand corner.

If Server Console UI is connected to multiple servers, select the required server in the tab list at the top of the window.

Filtering Events

The following filtering events are available:

Quick Filters

Quick filter options are made available to filter by Event Type, Status and Task Type. Simply select the values that you want to include in the filter (using Control - Select to select multiple items) and click on the Run Filter button on the bottom left of the screen to filter the events.

Free-text Filtering

Further free-text filtering options are available to filter by Project Name, Job Name, Task Name and User Name. These are free-text so that you can enter partial names into the fields. You can enter a partial name into any of these fields - provided the object contains the partial name, it will be displayed (though note that matching is case-sensitive). For example, if you use a naming convention where all projects working on live systems have a name including the word 'Live' you can display all events for live systems.

Date/Time Filters

The final set of filters, on the right-hand side of the screen, allow you to filter the list of events by date and time. A Date picker is provided to make it easier to specify a given date. Note that although only the most recent events are shown when accessing the Event Log, it is possible to apply filters to view older events if required.

Events may be filtered by their start times and/or by their end times. For example, you can apply a filter to see all Jobs and Tasks (but not System Tasks) that completed in the month of November 2008.

Column Selection

To change the set of columns that are displayed on the Event Log, click the Select Columns button on the top left of the Event Log area. The Select Columns dialog is displayed. Select or deselect the columns as required, and click OK to save or Cancel to abandon the changes. Alternatively, click Defaults to restore the default settings.

Note that Severity is a rarely used column - it is currently set to 50 for tasks or jobs that completed correctly, and 100 for tasks or jobs that raised an error or a warning.

Opening an Event

Double-clicking to open an event will reveal further detail where it is available.

Opening a Task will display the Task Log, showing any messages that were generated as the task ran.

For Jobs, if a notification email was configured on the job, the notification email will be displayed in a web browser when opening the completed event for the Job. Jobs with no notifications set up hold no further information.

Exporting Data from the Event Log

It is possible to export the viewable data in the Event Log to a CSV file. This may be useful if you are in contact with Oracle Support and they require details of what has run on the server.

To export the current view of events, click Export to CSV. This will launch a browser on the client for where to write the CSV file. Give the file a name and click Export to write the file.

1.2.2.5 Results

The Results window shows the staged data and staged results views of all jobs run from Server Console UI, and also the results of jobs run from the Command Line with a run label.

The window is divided into the Job History and Results Browser areas.

Job History

This area lists the jobs run in chronological order. It shows the Project, job, Run Label and end time for each one.

Results Browser

This area shows the details of the job selected in the Job History area above.

The Results Browser has various straight-forward options available as buttons at the top - hover over the button to see what it does.

However, there are a few additional features of the Results Browser that are less immediately obvious:

• Open in a New Window

• Show Characters

• Selecting Column Headers

• Purging Results

Open in a New Window

It is often useful to open a new window with the results from a given job. To do this, right-click on a job in the Job History area and select Open in a new window.

Show Characters

On occasion, you might see unusual characters in the Results Browser, or you might encounter very long fields that are difficult to see in their entirety.

For example, if you are processing data from a Unicode-enabled data store, it may be that the EDQ client does not have all the fonts installed to view the data correctly on-screen (though note that the data will still be processed correctly by the EDQ server).

In this case, it is useful to inspect the characters by right-clicking on a character or a string containing an unusual character, and selecting the Show Characters option. For example, the Character Profiler processor may work on some Unicode data, with a multi-byte character selected where the client does not have the required font installed to display the character correctly. The character therefore appears as two control characters.

If you right-click on the character and use the Show Characters option, EDQ can tell you the character range of the character in the Unicode specification.

The Show Characters option is also useful when working with very long fields (such as descriptions) that may be difficult to view fully in the Results Browser.

The Full column widths button Full column widths button will widen the columns to show the full data, but in this case there is too much data to show on the width of a screen. To see the FullDescription field as wrapped text, it is possible to right-click on the rows you want to view and use the Show Characters option. You can then click on the arrow at the top-right of the screen to show each value in a text area, and use the arrows at the bottom of the screen to scroll between records.

Selecting Column Headers

Clicking on the column header in the Results Browser will sort the data by that column. However, if you control-click on the column header (hold down the Ctrl key and click on the header), you can select all the visible (loaded) data in that column in the Results Browser. This is useful for example to copy all loaded rows, or to use them to create or add to reference data using the right-click option. Note that the Results Browser only loads 100 records by default, so you may want to use the Load All Data button before selecting the column header.

Multiple column headers can be selected in the same way.

Purging Results

To purge results in Server Console, right click on a Record in the Job History area.

Results can be purged by project, run label or job.

1.2.2.6 Result Purge Rules

Rules to automatically purge results under certain conditions can be set in Server Console using the Result Purge Rules dialog.

To open the dialog, select Tools > Purge Rules in the Server Console menu bar.

Adding a Rule

To add a rule:

1. Click the Add Rule button on the Result Purge Rules dialog. The New Rule dialog is displayed.

2. The Enabled checkbox is checked by default. To create a new rule without enabling it immediately, uncheck it.

3. Complete the fields as follows:

   1. Name - Enter a name for the rule. This field is mandatory.

   2. Purge results after - Specify the number of hours, days, weeks or months after which the results should be purged. Enter the number in the free-text field, and select the unit of time from the drop-down list. Alternatively, select "never" to ensure that results matching the rule criteria will not be purged. This field is mandatory.

   3. Project - If required, select a specific project for the rule.

   4. Job - If required, select a specific job for the rule.

   5. Run Label - Either specify an exact Run Label (by entering it, or selecting it from the drop-down list), or enter a regular expression in the Regex field to capture Run Labels containing specific terms. For example, the Regex .*test.* would capture all Run Labels containing the word "test".

4. Click OK to save the new rule, or Cancel to abandon.

Editing a Rule

To edit a rule:

1. Either double click the rule, or select it and click the Edit Rule button. A dialog with all the details of the rule is displayed.

2. Edit the fields as required.

3. Click OK to save the changes, or Cancel to abandon.

Enabling or Disabling a Rule

To enable or disable a rule, uncheck the Enabled check box next to it. This checkbox can also be edited when the rule is opened for editing.

Deleting a Rule

To delete a rule, select it in the Result Purge Rule dialog and click the Delete Rule button.

If a rule is deleted in error, click Cancel to close the dialog, not OK. On reopening, the accidentally deleted rule will be present again.

Setting the Rule Order

There are four buttons on the bottom right of the Result Purge Rule dialog for changing the order of the rules.

To move a rule, select it then:

• click the Move rule to top button to move it to the top of the list;

• click the Move rule up button to move it one place up the list;

• click the Move rule down button to move to one place down the list; or

• click the Move rule to the bottom button to move it to the bottom of the list.

1.2.3.1 Dashboard Elements

The My Dashboard view is comprised of Elements, each of which has a Status derived from the results of the Element.

A Dashboard Element is a line item of data quality information that a user can monitor on the Dashboard. There are four types of Dashboard Element:

• Indexes - a calculated value derived from a weighted set of Rule Results, tracked over time.

• Summaries - a summary of the statuses of a number of Rule Results

• Real Time Aggregations - an aggregation of the results of a Real Time Rule over a specified time period

• Rule Results - published results from a processor in EDQ

Indexes, Summaries and Real Time Aggregations are three different ways of aggregating Rule Results, which may be generated either in Batch or Real Time.

Indexes

An index is a type of dashboard element with a single numeric value, representing the aggregated results of a number of measures of data quality (Rule Results). The contributing measures are weighted to form an index across all chosen measures. Indexes are used for trend analysis in data quality, in order to track the data quality in one or many systems over a period of time. See the Dashboard Indexes topic for further information.

Summaries

A Summary is a type of dashboard element that aggregates a number of Rule Results into a summarized view showing the number of rules of each status (Red, Amber and Green).

Summary dashboard elements are created directly whenever publishing rule results from an EDQ process (where a summary is created summarizing all the rule results that are published from the process), or they may be configured manually by a dashboard administrator. Where configured by an administrator, the Summary may aggregate results from a number of different processes, and if required, across a number of different projects.

Note that unlike all other types of dashboard element, Summaries do not support trend analysis. This is because the Rule Results that comprise the summaries may be changed over time, and may be published at different times.

Real Time Aggregations

A Real Time Aggregation is a type of dashboard element that aggregates a single Real Time Rule Result dashboard element into a set of results for a different (normally longer) time period. Real Time Rule Results are published by processes that run in interval mode - normally continuously running processes. Intervals may be written on a regular basis so that EDQ users can see results on a regular basis - for example every hour, or every 100 records. However, it may be that Executives or other users may want to monitor results on a daily or weekly basis. This can be achieved by configuring a Real Time Aggregation and making this element available to users rather than the underlying Real Time Rule Results.

Rule Results

Rule Results are dashboard elements that directly reflect the results of an EDQ processor that is configured to publish its results to the Dashboard. Rule Results are therefore the most granular (lowest level) type of dashboard element.

Rule Results may be either Periodic (published from batch processes), or Real Time (published from real time processes that run in interval mode). The following icons are used to represent the different types of Rule Results in the Dashboard Elements pane of the Dashboard Administration window:

• Periodic Rule Results

• Real Time Rule Results

1.2.3.2 Dashboard Administration

Dashboard Administration allows an administrator to configure:

• the users who have access to published results;

• the way that published results are aggregated, into Summaries, Indexes and Real Time Aggregations; and

• the way that the status of each item on the Dashboard is calculated

It is also possible to delete items from the Dashboard, and to purge the results of published items.

See Dashboard Elements for further information on the terms and concepts used in Dashboard Administration.

Accessing Dashboard Administration

To access Dashboard Administration:

1. Open the Dashboard either from the Launchpad and log in as an administrator, or by right-clicking on a server in EDQ and selecting Display Dashboard.

2. Click the Administration button on the Dashboard front page.

This will start the Dashboard Administration, a Java Webstart application.

The Dashboard Administration GUI

There are two Views available on the Dashboard Administration GUI - Dashboard and Default Thresholds. These appear in the left-hand column:

• The Dashboard View allows you to administer all published results.

• The Default Thresholds View allows you to change the default way in which the status of Dashboard Elements of each type is calculated. The Default Thresholds can be over-ridden for specific Dashboard Elements if required.

Name: Name Validation (Daily)
Aggregation settings
Start date:  23-Jan-2009 00:00
Results by: Aggregate by time period: 1 days.

1.2.3.3 Dashboard Indexes

Indexes aggregate Rule Results, though it is also possible to aggregate indexes hierarchically to create an index of indexes. For example, a data quality index could be constructed for each of a number of source systems, or each of a number of types of data (customer, product etc.). An overall data quality index could then be constructed as an aggregation of these indexes.

Indexes are always configured in Dashboard Administration.

Index Calculation

The index value means little in isolation. However, as the score is calculated from the results of a number of executions of a process or processes (over time), trend analysis will allow the business user to monitor whether the index has gone up or down. This is analogous to monitoring the FTSE100 index.

A higher index value represents a higher data quality score. By default, a 'perfect' DQ index score is 1000.

Index of Rule Results

Where an index is made up of a number of Rule Results, it is calculated as a weighted average across the contributing results.

For example, a Customer Data DQ index may be made up of the following Rule Results and Weightings:

In this configuration, the Validate address and Validate name rules have the default weighting of 25% (a quarter of the overall weight across four rules), but the administrator has specified different weightings for the other rules – the Validate email address rule is interpreted as less important, and the Title/Gender mismatch as more important.

The actual index score is then calculated as a weighted average across internally calculated index scores for each contributing rule.

For each rule, an index score out of 1000 (or the configured base perfect score) is calculated as follows, where 10 points are awarded for a pass, 5 points for a warning, and no points are awarded for an alert:

(((# of passes * 10) + (# of warnings * 5)) / (# of checks *10)) * 1000

For example, if the results of the contributing rules are as follows:

The index scores of each contributing rule will be as shown below:

The overall index score is then calculated using the weightings, as follows:

Validate email address score (850) * Validate email address weight (0.125) = 106.25 +
Validate address score (800) * Validate address weight (0.25) = 200 +
Title/gender mismatch score (800) * Title/gender mismatch weight (0.375) = 300 +
Validate name score (800) * Validate name weight (0.25) = 200

The total Customer Data DQ index score is 806.25, and is rounded up to 806.3 for display purposes.

Index of Indexes

If an index is created to aggregate other indexes, the index is calculated simply as a weighted average of the contributing indexes. For example, the user might set up an index across a number of other indexes as follows:

If the index values of each indexes are as follows:

The index would be calculated as follows:

Customer data index (825) * Customer data index weight (0.50) = 412.5 +
Contact data index (756.8) * Contact data index weight (0.25) = 189.2 +
Order data index (928.2) * Order data index weight (0.25) = 232.5

The overall data quality index would have a value of 834.2.

Indexes of Staggered Audit Results

Indexes may aggregate results from a number of processes. Normally, it is expected that this form of aggregation will be used when the processes are executed at the same intervals. However, this cannot be guaranteed. In some cases, the processes contributing to an index will be out of step. For example, two data quality audit processes are executed. An index is configured to aggregate rule results from both processes, and results for the index history are published as follows:

This works by recalculating the results for the index every time one of its contributing processes is run. The results from the last run of each process are then used, and any previously calculated index results for a distinct date (day) are overwritten.

1.2.4.1 Match Review Summary Window

The areas of the window are described below.

Title Bar

Contains the name of the currently selected Review, a status bar showing the percentage of assigned review groups completed, and a direct link to launch the Review Application.

Reviews

Displays all the Reviews currently assigned (fully or partially) to the user.

Matching Status

This area breaks down the records by their matched status:

• Automatic Match

• Match

• No Match

• Possible Match

• Pending

Review Status

This area shows the records by Review Type:

• Awaiting Review

• User Reviewed

• No Review Required

Rules

This area displays every rule triggered during the Matching process, and the number of resolved and unresolved relationships identified by each rule.

1.2.4.2 Match Review Application Window

The Review Application is launched by clicking either:

• the Launch Review Application in the Title Bar; or

• any of the links in the areas of the Summary window.

This window is divided into the following areas:

Toolbar

This table describes the toolbar items:

Filter Groups

The fields in this area are used to search for specific groups using filtering criteria. See Filtering Groups for further information.

Records and Relationships Area

The Records area shows the records within the currently selected Review Group. Records that match are highlighted in yellow, records that are flagged for review are highlighted in mauve, and the currently selected record is always highlighted in blue.

The Relationships area shows the relationships between each record in the group, and indicates where there is a Match or a Possible Match.

So, in the example below, there are three records: R1, R2 and R3. The Relationship area shows that R1 is automatically matched with R2, and that there is a possible match between R1 and R3:

Review Merged Output

The Review Merged Output tab is divided into two areas:

• Records: The matched records in the currently selected Review Group.

• Merged Output: How the records will appear when merged.

1.2.4.3 Filtering Groups

The Filter Groups area is used to narrow down the groups displayed. Groups can be filtered by:

• searching for a text value in record attributes; or

• searching for specific review criteria; or

• both.

The following table describes the user interface elements in this area:

Filtering Examples

To search for an individual by family name:

1. In the Look For field, enter the family name, for example Williams.

2. Select Family Name in the Search In drop-down field.

3. Decide whether to search for the exact name or using case sensitivity, and check or clear the Case Sensitive and Exact Match boxes accordingly.

4. Click Find. The first group found is displayed in the Records and Relationships areas.

To search for groups by an individual family name and by a Match Rule name:

1. In the Look For field, enter the family name.

2. Select Family Name in the Search In drop-down field.

3. In the Relationship Attribute field, select the Match Rule Name.

4. Leave the Operator field set to =.

5. In the Relationship Value field, select the rule name, for example Exact Name, Postcode.

6. Ensure the Use OR logic box is checked, Check or clear the Case Sensitive and Exact Match boxes as required.

7. Click Find. The first group found is displayed in the Records and Relationships areas.

8. Navigate through groups returned using the Group Navigation buttons in the task bar.

Related Topics

Match Review Summary Window

Match Review Application Window

Reviewing Groups

Reviewing Merged Groups

1.2.4.4 Reviewing Groups

Selecting Groups for Review

The Match Review application allows users to review particular types of group by status, by the rule they trigger, or even by specific filter criteria.

In the Match Review Summary window, either:

• click Launch Review Application in the Title Bar to view all Groups; or

• click a link in the Matching Status, Review Status or Rules areas to view the Groups that fall within the category selected.

  Note:

  Most users will either need to view all Possible Matches (by clicking that link in the Matching Status area) or by Rule.

Alternatively, a user can search for Groups that fall within specific criteria relating to the Records within the groups or the Relationships between them. See the Filtering Groups topic for more information.

Applying a Decision

When the Groups are displayed, the user can navigate through them using the Group Navigation buttons in the toolbar of the Review Application.

To apply a decision, use the following procedure:

1. Review the information in the Records area. If necessary, click Highlight Differences to show where the Record Attributes differ.

2. In the Relationships area, select the required setting in the drop-down Decision field for each possible relationship. The options are:

   • Possible Match

   • Match

   • No Match

   • Pending

   Whatever option is selected, the Relationship is updated with the name of the user applying the decision, and the time the decision is made.

3. If required, proceed to the next group using the Group Navigation buttons.

Comments on Relationships

It is possible to add a comment to a Relationship, whether or not a decision has been made regarding it. The procedure varies depending on whether it is the first comment made or not

To add the first comment to a Relationship:

1. Click the Add First Comment button ( ) next to the Relationship.

2. In the Comment dialog, enter the text required.

3. Click OK to save (or Cancel to abandon).

4. The Comments dialog is displayed, showing the comment, the name of the user leaving the comment, and the date the comment was made.

5. Click OK to close the dialog. Alternatively, select the comment and click Delete to remove it, or click Add to add an additional comment.

To add an additional comment to a Relationship:

1. Click the Add Additional Comment button ( ) next to the Relationship.

2. In the Comments dialog, click Add.

3. In the Comment dialog, enter the text required.

4. Click OK to save (or Cancel to abandon).

5. The Comments dialog is displayed, showing the comment, the name of the user leaving the comment, and the date the comment was made.

6. Click OK to close the dialog. Alternatively, select the comment and click Delete to remove it, or click Add to add an additional comment.

Related Topics

Match Review Summary Window

Match Review Application Window

Filtering Groups

Reviewing Merged Groups

1.2.4.5 Reviewing Merged Groups

Once Records within Groups are identified as matches, they are merged together. It is then possible to review the results of the merge and, if necessary, override the way in which the merged output record has been generated.

Manual decisions made that override merged output are stored against a hash of the match group, and will be retained as long as the set of records in the match group stays the same. A group is marked as Confirmed in the UI if the Review Group from which the Match Group has been derived is fully resolved (for example, all Review relationships have been marked as either Match or No Match). Unless the source data or the match rules change, the set of records being merged is likely to stay the same and the manual overrides will apply.

When overriding output, values in the candidate records in the group (the set of records being merged) can be selected as the output attribute by right-clicking on the value and selecting the merged output field to populate with the value. Alternatively, it is possible directly to override the output value by typing into the merged output field.

Any errors in automatic merged output generation are highlighted to the user in the UI. An error indicates that a manual decision for the output field is required.

To begin reviewing Merged Groups:

1. Open the Match Review Application window.

2. Select the Review Merged Output tab. The Records and Merged Output areas are populated with the details of the first Merged Group.

3. The Records area lists all the records confirmed as matches. Click Highlight Differences to view where the Records vary from each other.

4. The Merged Output area displays the Merged Record resulting from the match.

   To manually override merged output, do any of the following:

   • From the Merged Output area, double click the attribute that you want to edit, and then enter the correct text.

     Tip:

     Right-click an attribute and select Clear to clear the value or select Reset to reset the attribute to the automatic output value.

   • Right-click a source data attribute value and select the merged output field to populate with this value.

     Tip:

     Source data attributes can often be identified by their colored backgrounds in the Records table; scroll right to find them. Identifier attributes that appear on the left side of the Records table have white backgrounds and cannot be used to populate merged output fields by right-clicking them.

5. Comments can be added and the Comment History for each Merged Record can be reviewed using the Latest Comment area to the right of the Merged Output area. Click the Add Comment button ( ) to add a comment or the Comment History button ( ) to view the history.

6. Edit the Merged Group as required, and navigate to the next group using the Group Navigation buttons in the toolbar.

Reviewing a Specific Merged Group

Sometimes it is necessary to immediately review the results of confirming a match. To review a specific merged group, use the following procedure:

1. Display the group in the Review area of the Match Review Application window.

2. Click Review Merged Output in the top-right of the Application window. The details of the currently selected Group are displayed in the Review Merged Output tab.

3. Review the Merged Group as required.

Related Topics

Match Review Summary Window

Match Review Application Window

Filtering Groups

Reviewing Groups

1.2.5.1 Case Management Concepts

This topic describes the major concepts used in the case management application. The terms covered here are:

• Alert

• Alert Key

• Attribute

• Case

• Case Key

• Case Source

• Data Source

• Derived State

• Extended Attribute

• Flag Key

• Parameter

• Permission

• Reception Rule

• State

• Transition

• Workflow

1.2.5.2 User Interface

The Case Management user interface is designed for simplicity and ease of use. All the Case Management screens use the same basic layout:

• The top of the screen contains a toolbar containing navigation and bulk edit controls;

• The left hand side of the screen contains summary information and editing controls;

• The rest of the screen contains more detailed information, which varies depending on the screen you are currently using.

• The lower right hand portion of the Details area contains a status bar which displays the currently connected server, the name of the logged in user and the version of Oracle Enterprise Data Quality that is in use.

There are four main screens:

• The Browser Screen, where you can browse cases and alerts according to predefined filters;

• The Filter Screen, where you can create and edit filters for use in the Browser screen;

• The Alert Screen, where you can see and change the details of a single alert;

• The Cases Screen, where you can see and change the details of a single case.

The exact contents and layout of your screens will depend on how Case Management has been configured, and also on the security permissions that have been granted to you.

1.2.5.3 Using Case Management

This section discusses the main operations performed in Case Management. It is divided into three main sections:

• Editing Cases and Alerts

• Working with Filters

• Export to Excel

State Changed On
Range
From: 26-Jan-2013 11:26
To: 27-Jun-2013 11:26

This will delete all items that match the currently configured filter.
 
Warning: If other bulk updates or deletes are occuring this action will not take place until after these and therefore the results of the filter may be different.

1.2.5.4 Case Management Administration

The Case Management Administration application is a part of the EDQ suite of applications. The three main areas of functionality provided by Case Management Administration are:

• Workflow Administration

• Case Source Administration

• Permission Administration

caseData.caseType=='issue' &&
updateKeyChanged=='true' &&
caseData.currentState=='Close Account Required'

1.2.5.4.10 Example Workflow - 4-Eyes Review Process

This workflow is provided as an example of how Transitions and Conditions can be used to enforce a strict review hierarchy. It requires that at least two distinct individuals must have reviewed an Alert before a final decision is made. The diagram below illustrates this:
Description of the illustration ''case_man_admin_4eyeprocess.png''

As shown above, User 1 can mark Alert 1 as a Suspected False Positive, and therefore cannot mark it as a Confirmed False Positive. Only another user (User 2 in this case) can do this.

Similarly, if User 2 escalates Alert 2 to a Suspected False Positive, only User 1 can mark it as a Confirmed False Positive.

Below is a screenshot of how a simple 4-Eyes workflow may appear in the Workflow Editor. The three possible States (Open, Suspected False Positive, Confirmed False Positive) and Transitions (Suspect false positive, Confirm false positive and Reopen) are shown in their respective lists:

Description of the illustration ''case_man_admin_4eye_workflow_665x256.png''

To enforce the 4-eye rule, the Transitions must be configured as follows:

• No restrictions are required for the Suspect false positive Transition.

• Confirm false positive - Add the Suspect false positive Transition to the Blocking Transitions field. This prevents a user applying the Confirm false positive Transition if they made the Suspect false positive Transition.

• Reopen - Add the Confirm false positive Transition to the Clear Blocking Transitions field. This clears any restrictions on the Confirm false positive Transition being applied to the Alert, for all users.

1.3.1.1 Add Message ID

The Add Message ID processor adds a field containing a numeric identifier in to a process. Message IDs identify a packet of data, known as a message, as it passes through an EDQ process.

Add Message ID is used to add a numeric identifier to records. The message ID is unique within a data stream for records in batch processes and records created by real-time processes with single record inputs.

Note that, for a given data stream, this identifier will not be unique in the following circumstances:

• The Message ID will not be unique for records originating from real-time processes or web services which have multi-record inputs. In this case, the message ID will be the same for all the records originating from a single request.

• The Message ID will not be unique if records within a stream are split by the process. In this case, the message ID will be the same for all records derived from a single originating record.

The Add Message ID processor presents no summary statistics on its processing.

In the Data view, the added output attribute is shown.

Example

In this example, the message ID has been added to a set of records in a batch process. The process has been run in batch mode, and does not split records, so in this case, the message IDs are all unique:

1.3.1.2 Add User Details

The Add User Details processor adds in to a process details of the authenticated EDQ user calling the Web Service or Job where the process is used.

Add User Details is normally used in applications that use EDQ Web Services for data validation, cleansing or matching. Capturing the details of the authenticated user in the EDQ process is useful for two reasons:

• To keep a full audit trail of all records processed including details of the user that entered/updated the record

• So that the process can use the user details, for example to send an email to the authenticated user's address with the results of data processing

For example, where EDQ is used to screen individuals against watchlists, an email may be sent out to the authenticated user of the individual screening service with the result of the matching process, using the email address stored for the user in EDQ (as set up by an administrator in User Configuration). Additional user details, such as organization and phone number, may be added to the audit trail so that complete information on the user submitting the record is retained.

1.3.1.3 Expression

The Expression processor allows you to write an expression which may encompass the logic of a number of other EDQ processors and other functions, using an abbreviated expression language.

Expressions are useful where the required logic of a processor can be expressed most logically and succinctly by writing an expression, rather than by configuring several processors. This is commonly the case when performing mathematical operations, for example.

Inputs

The Expression processor may take any number of inputs, of any type, with the exception of formal Date attributes.

Options

See Expression Notes for detailed notes on EDQ's expression language.

Outputs

Data Attributes

Flags

None

Execution

Results Browsing

The Expression processor presents no summary statistics on its processing.

In the Data view, each input attribute is shown with the ExpressionResult attribute to the right.

Output Filters

None

1.3.1.4 Expression Filter

The Expression Filter processor allows you to write an expression using EDQ's expression language, which will be used to pass or fail records and split them into different output filters.

The Expression Filter processor is effectively an advanced form of the Logic Check processor, which provides a GUI to construct the expression. As the Logic Check GUI shows the expression as it is constructed, it is therefore possible to learn how valid expressions are constructed using Logic Check.

Expression Filter is useful where you want to apply a simple test to all input records, and split out records that pass the test from records that fail it.

Inputs

The Expression Filter processor may take any number of inputs, of any type, with the exception of formal Date attributes.

Options

See Expression Notes for detailed notes on EDQ's expression language.

Outputs

Data Attributes

None

Flags

Execution

Output Filters

The following output filters are available from an Expression Filter processor:

• True - that is, records that passed the expression filter test

• False - that is, records that failed the expression filter test

1.3.1.5 Generate Warning

The Generate Warning processor is designed to generate a warning, and/or trigger a process failure, in the event of processing a certain number of records (over a configurable threshold).

Use the Generate Warning processor in conjunction with the notification capabilities in Jobs if you want to be made aware of any warnings when running processes. For example, you might use an audit processor to check for data validity, and you might want to raise a warning if >n records fail the check. You might even want to fail the process to ensure that you don't continue running, if you know the remaining parts of the process need to be rerun in any case.

The Generate Warning processor therefore allows you to set your own definition of what constitutes a 'data error' that you want to see in the Event Log and in any notification emails.

Generate Warning and Intelligent Execution

To avoid generating unnecessary warnings when running the same process repeatedly while designing it, the Generate Warning processor only generates a new warning if its rerun marker is set, or if run in a process or job with intelligent execution turned off.

The rerun marker will always be set if a new set of data is being input to the process, or if the configuration of the processor is changed.

1.3.1.6 Message Handling Script

The Message Handling Script processor allows you to specify your own processor logic using JavaScript or Groovy, entered as an option for the processor.

The Message Handling Script processor is closely related to the Script processor. The main difference between this processor and the Script processor is that the Message Handling Script processor is designed to handle messages, which may contain many records. The Message Handling Script processor is capable of performing operations over all the records in a message, and has access to message level attributes such as the message ID.

Use the message handling script processor to define processing logic that acts across all the records in a message, or that acts selectively on only the first record in the message.

The Message Handling Script processor presents no summary statistics on its processing.

In the Data view, each input attribute is shown with the output attributes to the right.

Script

The following notes are important to consider when writing a script for the Message Handling Script processor.

The Message Handling Script processor takes a message as its unit of work. A message may be composed of multiple records, which are modeled as an array named records within the Message Handling Script processor. Each record in the array has its own set of input attributes and its own output attribute, corresponding to the input1 and output1 attributes of the standard Script processor.

In addition, a pre-defined variable, messageid, contains the unique message ID, and a variable tags, which contains the message level tags.

Each record also has a cancel() function, which suppresses further processing of the record in the process.

To set a function include the line:

#! function : doit

at the top of the script (where 'doit' is the function name; change this to the function name you are using).

To change the language of the script to groovy rather than Javascript, add the line:

#! language : groovy

Examples

The following script iterates across all the records in a message to calculate the sum of the values of the first input attribute for each record (which is assumed to be a number). It then iterates across the records a second time, and sets the value of the output attribute of each record to the sum calculated in the first step:

var sum = 0;
for (var i = 0; i < records.length; i++)
{
  sum += records[i].input1[0];
 
}
 
for (var i = 0; i < records.length; i++)
{
  records[i].output1 = sum;
 
}

The following script computes the sum as before, but all but the first record are suppressed using the cancel() function. This results in a single record - the first - being output per message, with the total of the first input attribute across all records as its output attribute.

var sum = 0;
for (var i = 0; i < records.length; i++)
{
  sum += records[i].input1[0];
 
}
 
for (var i = 0; i < records.length; i++)
{
  if (i == 0)
 
  {
 
    records[i].output1 = sum;
 
  }
 
  else
 
  {
 
    records[i].cancel();
 
  }
 
}

1.3.1.7 Script

The Script processor allows you to specify your own processor logic for a simple processor using JavaScript or Groovy. The script is entered as an option on the script processor.

Note that it is possible to use a script in a processor that is created in EDQ, so that the script is not easily accessible by the end user of the processor. It is also possible to use script when writing a new processor to add into the processor library - this allows for greater complexity in the script - for example to enable multiple inputs and outputs, options that use Reference Data, and results views.

Use the script processor to define some simple processing logic that cannot be achieved easily using any of the provided processors.

Inputs

Any number of attributes of any type may be input to a Script processor. The selected input attributes are mapped to a JavaScript array named input1.

Options

To set a function include the line:

#! function : doit

at the top of the script (where 'doit' is the function name; change this to the function name you are using).

To change the language of the script to groovy rather than Javascript, add the line:

#! language : groovy

Outputs

The Script processor supports a single output data attribute only. The type of this attribute is determined by the setting of the Result type option.

The single output must be assigned to the script name output1.

Data Attributes

Flags

None

Execution

Results Browsing

The Script processor presents no summary statistics on its processing.

In the Data view, each input attribute is shown with the output attributes to the right.

Output Filters

None

Examples

output1 = java.util.UUID.randomUUID().toString()

var res = '';
for (var i = 0; i < input1.length; i++)
  { if (i > 0) res += '||';
     res += input1[i];
  }
output1 = res;

Script execution failed: TypeError: Cannot set property '0.0' of null to '<value>' ([script]#2)

var output1 = new Array();

1.3.2.1 Business Rules Check

The Business Rules Check processor enables users to check data against a set of business rules, which can be defined and maintained outside of EDQ. Validation errors can be output for each record checked.

The Business Rules Check processor is used to apply potentially complex validation rules to data in a single processor. The business rules can be defined in reference data or in an external file, allowing the rules to be maintained externally to EDQ.

The following tables describe the configuration options:

If an Excel file is used to define the business rules, it must be placed in the [Install Path]/oedq_local_home/businessrules directory, where [Install Path] represents the root of the EDQ installation, and must conform to the format specified in the Defining Business Rules topic.

Specifying Identifiers

Identifiers are used to map the input attributes of the Business Rules Check processor onto the attribute tags referenced in the business rules document. The mappings between identifiers and attributes are defined on the Identify tab of the Business Rules Check processor dialog.

For each identifier, select the appropriate input attribute from the drop-down on the right.

• If an identifier which has no mapped input attribute is used in a rule, that rule will be ignored.

• If an identifier with no mapped input attribute is used in a condition, the condition is assumed to evaluate to TRUE.

This processor always appears with a re-run marker, indicating that it will be completely re-executed each time the process is run, regardless of whether or not its configuration has changed. This will also mean that processors that are downstream of the processor will need to be rerun. This is because there may be changes made outside of the EDQ application that could lead to different results on subsequent executions.

The following table describes the statistics produced by the profiler:

Output Filters

The following output filters are available from a Logic Check:

• Records which passed the business rule check

• Records which failed the business rule check

Example

In this example, the user wants to use an Excel file called contactDetailsCheck.xls to validate some of the contact details in their customer data. This file contains rules which check that:

• The surname field contains some data;

• The surname contains only letters;

• That at least one of Town and County is specified in the address;

• That the postcode is valid;

• That the telephone number is valid.

The summary data:

A drill down on records failing the check shows how those records break the rules, and displays the error codes, severities, and error messages defined as part of the business rules configuration.

1.3.2.2 Cross-attribute Check

The Cross-attribute Check processor allows you to compare the values in two attributes in order to check the consistent application of a business rule.

Note that you can only compare attributes of the same Data Type; that is, a String attribute must be compared with another String attribute, a DATE with another DATE, and a Number with another Number.

Use Cross-attribute Check to check that the data for two related attributes is correct. For example, you may want to check that Date_Of_Death values are later dates than Date_Of_Birth values.

The following table describes the configuration options:

The following table describes the statistics produced by the profiler:

Output Filters

The following output filters are available from a Cross-attribute Check:

• Pass; that is, records where the comparison passed

• Fail; that is, records where the comparison failed

• Null comparison; that is, records with a null value in either or both attributes checked

Example

In this example, two attributes are compared: DT_ACC_OPEN and DT_PURCHASED, to see how many have repeated data:

You can drill down on the Records Passing Comparison:

1.3.2.3 Data Type Check

The Data Type Check processor checks that the values in a String attribute conform to a consistent data type, and categorizes as invalid any records with values that are not of the expected data type.

Note that Number and Date attributes are by definition 100% consistent with regard to their data type, and so cannot be checked.

The Data Type Check is a useful way of quickly finding values that have been entered into the wrong fields in a user application - typically numbers or dates that have been entered into fields that expect text values only.

Note that it is possible to 'expect' dates or numbers in a String attribute, and categorize as invalid any values that are not of the expected type. This is provided because dates and numbers are not always held in attributes with a controlled data type that can be read from the schema of the data source.

The following table describes the configuration options:

The following table describes the statistics produced by the profiler:

The Date Formats Reference Data used by the Data Type Check must conform to the standard Java 1.6.0 or later SimpleDateFormat API.

Clicking on the Additional Information button will show the above statistics as percentages of the total number of records analyzed.

Output Filters

The Data Type Check produces the following output filters:

• Valid records

• Invalid records

Example

In this example, the Data Type Check is used to check that all values for a NAME attribute are in textual format.In this case, null values are treated as invalid.

In this case, a date value that was entered into the wrong field, and two Null Name values were found.

1.3.2.4 Duplicate Check

The Duplicate Check processor provides a simple way of checking for duplicate values across either one or many attributes.

Use the Duplicate Check to identify any duplicate values that may cause a problem for a data migration (for example, in key attributes), or as an initial check for duplicate records in the data.

The following table describes the configuration options:

The Duplicate Check assesses duplication across a batch of records. It must therefore run to completion before its results are available, and is not suitable for a process that requires a real time response.

When executed against a batch of transactions from a real time data source, it will finish its processing when the commit point (transaction or time limit) configured on the Read Processor is reached. The statistics returned will indicate the number of duplicates in the batch of transactions only.

The following table describes the statistics produced by the profiler:

Output Filters

The following output filters are available from a Duplicate Check:

• Duplicate records

• Non-duplicate records

Example

In this example, the Duplicate Check processor is used to look for duplicate company names in a BUSINESS attribute:

You can drill down on duplicated values:

1.3.2.5 Email Check

The Email Check processor checks for Email Addresses in an attribute, and validates them syntactically; that is, that the address conforms to a correct pattern for Email Addresses.

Note that Email Check uses a reference list of regular expressions to check for valid email addresses. You may change the rules used by the Email Check by modifying the list of regular expressions used for the check.

Use the Email Check processor to validate that email addresses have been entered correctly.

The following table describes the configuration options:

The following table describes the statistics produced by the profiler:

Output Filters

The following output filters are available from an Email Check:

• Valid records

• Invalid records

Example

In this example, Email Check is used to check data in an EMAIL attribute:

You can drill down on valid or invalid values.

Invalid values:

1.3.2.6 Invalid Character Check

The Invalid Character Check processor provides a quick and easy way to find values that contain odd characters.

Use the Invalid Character Check to check for unusual characters. This is particularly useful when analyzing free text fields, which may have 'data cheats' in them, where data entry users have worked round mandatory fields by entering dummy characters such as #. The Invalid Character Check is also useful for finding typos.

If the invalid characters do not signify anything, they can simply be removed by adding a Denoise processor.

The following table describes the configuration options:

The following table describes the statistics produced by the profiler:

Output Filters

The following output filters are available from a Invalid Character Check:

• Valid records

• Invalid records

Example

In this example, a NAME attribute is checked for invalid characters such as ()#%^*$£"!A number of records are found containing the # character.

You can drill down on invalid values:

1.3.2.7 Length Check

The Length Check processor provides a quick and easy way for checking an attribute for values of the appropriate length.

The Length Check can check either, or both, of the following:

• The total length in characters (including whitespace and control characters)

• The number of words

You can choose the way 'words' are counted using options on the Length Check. By default, words are separated by spaces. For example, the word count of 'Oracle Limited' is 2.

Use the Length Check to ensure that the data within the attribute will meet either its technical or business purpose. For example, if migrating an attribute's data to a shorter attribute in a target system, you may choose to truncate the data, and then check that it conforms to the character length restrictions of the target field before migrating. Alternatively, there may be a business reason why a value should not be over a set number of characters, or words. For example, you might want to check a Surname attribute for all values over 2 distinct words in length, as this might indicate misuse of the attribute - for example to store a Company Name value.

The following table describes the configuration options:

The following table describes the statistics produced by the profiler:

Click on the Additional Data button to see the above statistics as percentages of the number of records analyzed.

Output Filters

The following output filters are available from a Length Check:

• Valid (records where both counts were valid)

• Invalid (records where both counts were invalid)

• Invalid character count (records with an invalid character count, but a valid word count)

• Invalid word count (records with an invalid word count, but a valid character count)

Example

In this example, Length Check is used to check the length of an Account Number attribute (CU_ACCOUNT) for any values with a character count not in the range 10-11, and any values that do not consist of a single word:

You can drill down on a bad character, good word lengths count.

Note that the CU_ACCOUNT attribute for the above records is too short.

1.3.2.8 List Check

The List Check processor checks the data in an attribute against reference lists of valid and invalid values for the attribute.

The List Check allows case sensitive or insensitive matching, and can match the reference lists in one of a number of different ways:

• Contains (the value must contain a matching list entry)

• Whole value (the value must match the list exactly)

• Starts with (the value must begin with a matching list entry)

• Ends with (the value must end with a matching list entry)

• Delimiter match (match the list after splitting the data using the specified delimiters)

The List Check is an important processor, used in auditing to find valid and invalid values in a data attribute. Use the results of the Frequency Profiler or Phrase Profiler to create lists of valid and invalid values, and use them in a List Check to audit the data against the lists on an ongoing basis.

List Check allows the use of up to two reference lists - a list of Valid values for the attribute, and a list of Invalid values.

You may choose only to use one of these two lists. For example, if you discover from profiling that there are many different valid values for an attribute, you may want only to check the attribute for invalid values, and consider the non-matching values as either Valid, or Unknown - for example simply to look for suspect words such as 'Test' in a Surname attribute.

If, however, the attribute has a small number of valid values, you may want simply to check the data against a list of valid values, and consider the non-matching values as Invalid, or Unknown - for example to check Title values against a small set of valid titles.

Finally, you can use both lists, and recognize both valid and invalid values, with values that do not match either list categorized as Unknown.

The following table describes the configuration options:

The following table describes the statistics produced by the profiler:

Drilling down on any of the above statistics reveals a count of the distinct values that were found to be Valid, Unknown or Invalid. You can then drill down again to see the records themselves.

Output Filters

The following output filters are available from a List Check:

• Valid records

• Unknown records

• Invalid records

Example

In this example, the List Check is used to check the values in a Business attribute using lists of valid and invalid Business values collated from Frequency Profiling.

Note that values that do not match either list are categorized as Unknowns.

A summary view:

A drill down on unknown records:

A drill down on invalid records:

1.3.2.9 Logic Check

The Logic Check processor enables users to perform a logic check, using multiple criteria to route or filter records.

The Logic Check processor is normally used as a way to filter records to a required set. For example, if all records are passed through a number of checks, it may be useful to add a Logic Check at the end to select which records are considered as overall passes and fails, using the Flag attributes added by other audit processors.

Logic Check may also be used to filter down records to a specific set where the data matches a number of criteria across a number of attributes.

The following table describes the configuration options:

The following table describes the statistics produced by the profiler:

Output Filters

The following output filters are available from a Logic Check:

• Records with data in the checked attributes

• Records without data in the checked attributes

Example

For example, the user wants to filter the records which have all of the following attributes populated: NAME, ADDRESS1, POSTCODE, EMAIL. These attributes are profiled in the Quickstats profiler first to generate the Populated flags.

A summary view:

You can drill down on the records passing check, or records failing check.

1.3.2.10 Lookup Check

The Lookup Check processor allows you to check for records in a set of Reference Data that are related to those that you are currently working with, for example, data from another table in a relational database, or related data in a separate system.

Lookup Check uses an exact match (using one or more key attributes) to match records in the Reference Data.

Use Lookup Check when you want to check how many related records for each of your working records exist in a Reference Data table. The Check can fail any records that have either too many, or too few, related records, according to configurable options.

For example, you may want to check that there is at least one Address record for each of your Customer records.

The following table describes the configuration options:

When looking up external data (that is not staged), the appropriate level of performance of the lookup will depend upon there being appropriate indexes on the lookup columns for the selected Reference Data. Also, if looking up external reference data, the Lookup Check processor will always appear with a re-run marker, indicating that it will be completely re-executed each time the process is run, regardless of whether or not the actual processor configuration has changed. This will also mean that processors that are downstream of the Lookup Check processor will need to be rerun. This is because EDQ cannot detect whether or not the external reference data has changed, and therefore has to assume that it has changed (since external lookups are usually used for dynamically changing reference data), and so re-executes the lookup in order to ensure the consistency of dependent results.

The following table describes the statistics produced by the profiler:

Output Filters

The following output filters are available:

• Valid records

• Invalid records

Example

In this example, a Lookup Check is performed to check that at least one order (record in the Workorder table) exists for each Customer record. All Customers without any orders may then be tagged as 'Prospects' and not included in the active Customer statistics:

A summary view:

A drill down on invalid records:

1.3.2.11 No Data Check

The No Data Check processor provides a simple way of checking whether or not there is any meaningful data in an attribute, or across a number of attributes.

The No Data Check checks not only for null values, but also for empty Strings, and values consisting entirely of spaces or non-printing characters - though only null values will exist if the data has already undergone No Data Handling in the snapshot.

If a record has any data other than white space in any of the attributes analyzed, it is categorized as containing data. Otherwise, if all attributes analyzed contain No Data, it is categorized as containing no data.

Use the No Data Check processor to check for complete and incomplete values in an attribute, or across a number of attributes. For example, you may want to check for all records that have a missing Gender value, and add it where possible using the data available, perhaps by mapping from a Title attribute. Or, you may want to isolate all records that have no data at all in a number of Name or Address fields.

The following table describes the configuration options:

The following table describes the statistics produced by the profiler:

Output Filters

The following output filters are available:

• Records with data in the checked attributes

• Records without data in the checked attributes

Example

In this example, the No Data Check is used to find records without an email address (in an EMAIL attribute):

A summary view:

A drill down on Without Data:

1.3.2.12 Pattern Check

The Pattern Check processor checks the pattern of data in an attribute against reference lists of valid and invalid patterns.

Use the Pattern Check processor to ensure that the data in an attribute conforms to one of the valid patterns for that attribute. Data may need to conform to a set of valid patterns for either technical or business reasons. For example, when migrating data, the target system may require that all data for a given attribute consists of numeric characters only, and with minimum and maximum length restrictions. Alternatively, for business reasons, it may be that you want to tag as invalid records that have bad data, or data in the wrong attributes, for example, numeric values in Name fields, malformed product codes etc.

You can create the lists of valid and invalid patterns from the data itself using the Patterns Profiler.

Pattern Check allows the use of up to two reference lists - a list of Valid patterns for the attribute, and a list of Invalid patterns.

You may choose only to use one of these two lists. For example, if you discover from profiling that there are many different valid patterns for an attribute, you may want only to check the attribute for invalid patterns, and consider the non-matching values as either Valid, or Unknown.

If, however, the attribute has a small number of valid patterns, you may want simply to check the data against a list of valid patterns, and consider the non-matching values as Invalid, or Unknown.

Finally, you can use both lists, and recognize both valid and invalid patterns, with values that do not match either list categorized as Unknown.

The following table describes the configuration options:

The following table describes the statistics produced by the profiler:

Drilling down on any of the above statistics reveals a count of the distinct patterns that were found to be Valid, Unknown or Invalid. You can then drill down again to see the records themselves.

Output Filters

The following output filters are available:

• Valid records

• Unknown records

• Invalid records

Example

In this example, Pattern Check is used to verify values in an Account Number attribute (CU_ACCOUNT) using lists of valid and invalid patterns collated from Patterns Profiling.

Values that did not match either list of pattens were categorized as Unknown.

A summary view:

A drill down on invalid records:

1.3.2.13 RegEx Check

The RegEx Check processor checks the data in an attribute against reference lists of valid and invalid regular expressions for the attribute.

The case-sensitivity and matching technique (Whole Value / Contains / Starts With / Ends With) of the check can be controlled.

The RegEx Check processor is a powerful tool, allowing you to validate data according to its exact content, using the position of data, partial and exact values, and wild cards.

The RegEx Check is useful in order to check any data that should be in a consistent structure, for example, UK National Insurance Numbers.

Regular Expressions

Regular expressions are a standard technique for expressing patterns and manipulating Strings that is very powerful once mastered.

Tutorials and reference material about regular expressions are available on the Internet, and in books, including: Mastering Regular Expressions by Jeffrey E. F. Friedl published by O'Reilly UK; ISBN: 0-596-00289-0.

There are also software packages available to help you master regular expressions, such as RegExBuddy, and online libraries of useful regular expressions, such as RegExLib.

The following are some examples of regular expressions that could check data:

The following table describes the configuration options:

The following table describes the statistics produced by the profiler:

Drilling down on any of the above statistics reveals a count of the distinct patterns that were found to be Valid, Unknown or Invalid. You can then drill down again to see the records themselves.

Output Filters

The following output filters are available:

• Valid records

• Unknown records

• Invalid records

Example

In this example, a RegEx Check is used to check the format of an Account Number attribute (CU_ACCOUNT), using a Whole Value match against the following regular expression:

^([0-9]{2})(-)([0-9]{4,5})(-)([a-zA-Z]{2})

This regular expression dictates that values must start with exactly 2 digits, following by a hyphen, followed by either 4 or 5 digits, followed by another hyphen, followed by two letters.

A summary view:

A drill down on invalid records:

1.3.2.14 Suspect Data Check

The Suspect Data Check processor checks an attribute value for a variety of common data entry 'cheats', such as entering 'aaa' in a name field.

Specifically, it can check for any or all of the following:

• Repeating alphabetic characters (for example, 'aaa')

• Repeating numeric characters (for example, '111')

• Repeating non-alphanumeric characters (for example, '>>>')

• Repeating patterns (for example, 'abcabc')

• Minimum character length (for example, for short values such as 'x')

Use the Suspect Data Check to check for common user ruses which result in suspect data in compulsory columns.

Where an empty value cannot be entered at the point of data entry, the user may enter a single character - for example a space, a full stop, or a random single character - to get past that point.

Alternatively, a single but repeating character may be entered - for example '9999' - or perhaps a repeating pattern of characters - for example 'asdasd'.

Note that this may not be the fault of the user, but of the business process and/or supporting applications. The user may not know or have available the complete set of data that the data entry application is requesting, but is required to enter the data into the system.

The following table describes the configuration options: