Data Discovery Options can be access by clicking the expand button
on the Discover section of the main menu on the Data Discovery tab:
Options
Allow SELECT *
Selecting data from a relational database with the * wildcard is usually a bad practice. It's even a worse practice when using DSF to discover data.
By default DSF will NOT evaluate any SQL Queries that do not explicitly select the field names from the database tables. Any Views or queries with the "SELECT *" will be ignored.
The * wildcard will include ALL fields found in the tables using their native database names. This is not recommended and should only be enabled if there is no other alternative.
Report Detailed Errors
The wait time (in seconds) before terminating the attempt to execute a query sent to your database server before generating an error.
This setting should only be changed if queries sent to your database server take a very long time to run.
Test Database Views Before Discover
During data discovery, it's bad if the system tries to discover data from database views that are broken and do not run properly. However, testing every view before discovery can slow down the discovery process.
Most database do not have broken views so DSF assumes they all work by default and does not test them first.
If, after discovery, you find Entities that report errors when creating Views, there is a strong possibility that the Entity was derived from a bad database view. The best thing to do is delete the Entity find the offending database View and fix it or exclude it. Then Re-Discover the Data Source.
If you don't mind having longer discovery times because you are unsure of the validity of your database views, then enable this option. ONLY database views that properly run will be discovered.
Track Database Changes
This option will make a detailed image of your physical database structure after any discovery of data. From that point on, if your physical database structure changes (Tables added or modified) DSF can report on those changes for you.
If changes did occur you will have the opportunity to include those changes into your DS Framework.
Notify When Database Changes
When enabled you will be notified on the main screen when database changes have occurred since your last discovery.
Regular Expression pattern.
You can test your Regular Expression patterns here.
Regular Expression entries MUST be prefixed with 'REGX:'
The characters following the prefix is the actual Regular Expression pattern used.
Regular Expression example exclusion entry:
REGX: ^\d{8}_ This excludes any name that starts with 8 digits followed by an underscore character and then any number of characters following that. Example Value: 20180102_DailyRun
Only use Regular Expressions when necessary. Regular Expressions can easily exclude ALL entries by accident.
Other examples:
REGX: (_\d{8})(?!.*\d) This excludes values ending with underscore followed by 8 digits. Example Value: DailyRun_20180102
REGX: (\d{14})(?!.*\d) The excludes values ending with 14 digits. Example Value: MyTableName20180123105219
Data Types
Some database have Data Types that may have no real value to the user. They can be excluded by adding the Type Name to the exclusion list.
Exclusion List (Defaults)
Image
RAW
Long Raw
Bfile
blob
clob
nclob
MediumBlob
xml
By default DSF adds the Data Types list above as exclusions.
Tables
Enter table names here on the exclusion list that you know you don't need to discover.
Some databases include tables with dates or timestamps in their names. You could have hundreds or even thousands of these in your database. If this is the case the structure of the tables are most likely the same. DSF only needs to discover one of them to include the necessary data elements. It would be a complete waste of time for DSF to repeat the process hundreds of times over and over only to get the same results in the end.
Use a Regular Expression to exclude the unwanted table names. See the examples above.
Views
Views are just like tables and some systems repeat the same names over and over. Treat view exclusions the same as table exclusions above.
Schemas
Some databases separate data using schemas. Some schemas add no value to your reporting needs so why discover them. Exclude unnecessary schemas the same way you would tables or views above.