Vocabulary_ID=17
The following describes the procedure to update
the READ vocabulary with the
latest release of the source.
The
load process is a complete refresh of data. All new vocabulary
entieties will be added. All vocabulary entities that do not exist in
the latest files set will be marked as deprecated.
1.1 Scripts Transfer
All scripts and files listed at the bottom of this page should be transferred to specially created directory (eg /data/Read).
Script Name |
Description |
17_create_schema.sql |
Create prestage-stage schema with tables and indexes
|
17_JJ_vocab_17.zip | JJ mapping sample data file |
17_JJ_vocab_17.ctl | JJ mapping control file |
17_transform_row_maps.sql |
Convert and store in stage table SOURCE_TO_CONCEPT_MAP_STAGE |
17_load_maps.sql
| Load only new maps into SOURCE_TO_CONCEPT_MAP table, add invalid code information |
1.2 Download data from source
1.2.1 Download data from JJ source
Usually a current internaitonal release is sent by e-mail by the analyst of the company JJ, responsible for the data.
File sample JJ_vocab_17.zip can be downloaded in the section «Add files» at the end of this page
Note: Make
sure you perform updates in chronological order. So, if you want to
process a January and April release, execute the below steps for
January, and after successful completion, for April.
Download the latest release file (e.g. JJ_vocab_17.zip)
Transfer downloaded zip file to /data/Read directory.
Latest production zip file is ~1,6MB in size
Note: For the rest of the document we will be referring to current schema as READ_20140131 you would have to adjust this entry to correspond to current release date file.
1.3 Build Database Schema
All tables will be located in the schema READ_<DateOfSourceFile> as following: READ_20140131
The following steps will create a schema and
will grant to a schema owner appropriate permissions.
In addition to the schema, it will create table to
load raw data: JJ_VOCAB_17 as well as staging tables: CONCEPT_STAGE, CONCEPT_ANCESTOR_STAGE,
CONCEPT_RELATIONSHIP_STAGE,
SOURCE_TO_CONCEPT_MAP_STAGE
$ sqlplus System/<SystemPass>@DEV_VOCAB @17_create_schema.sql <READ_Schema> <Pass_READ_Schema>
Format of READ_Schema should be READ_YYYYMMDD (e.g. READ_20140131 )
Staging tables match Vocabulary schema version 4.0 with slight modification. Modifications made to the tables are:
CONCEPT_STAGE.CONCEPT_ID is nullable
SOURCE_TO_CONCEPT_MAP_STAGE
.SOURCE_TO_CONCEPT_MAP_ID is nullable
CONCEPT_RELATIONSHIP_STAGE
.REL_ID is nullable1.4 Raw data load into Oracle tables
1.4.1 Extract files from the zip archive into the current directory
$ unzip -u JJ_vocab_17.zip
1.4.2 Verify that the following file have been created:
JJ_vocab_17.csv (Latest file size ~4,7MB)
1.4.3
Load FDA raw data into
Oracle.
This sqlldr process will run for about 1min.
$ sqlldr READ_20140131/myPass@DEV_VOCAB control=17_JJ_vocab_17.ctl
1.4.4 Verify that files with extension .bad are empty, no records have been rejected.
1.4.5 Copy data and intermediate files to backup area. In this step, downloaded files will be archived to preserve source of the data. Discuss a location of backup area with your system administrator.
$ cp -Ru /data/READ/*.zip /data/backup_area ; cp -Ru /data/READ/*.log /data/backup_area
1.4.6 Verify that number of records loaded is equivalent to prior production load
$ sqlplus READ_20140131/myPass@DEV_VOCAB
Execute the following SQL commands:
SELECT count(*) FROM JJ_VOCAB_17; |
Current production row count is: 8,3764 Rows
1.5 Loading Staging Tables from raw
1.5.1 Convert and store in
staging table maps
Loaded from raw staged data JJ_VOCAB_17 into the staged data SOURCE_TO_CONCEPT_MAP_STAGE
$ sqlplus READ_20140131/myPass@DEV_VOCAB @17_transform_row_maps.sql
1.5.2 Verify that number of records loaded is equivalent to prior production load
In this step we will verify that number of active records in staging
tables is valid prior to transfer data to historical DEV schema.
a. Number of Records in stage table
b. Number of records in DEV schema not deleted (active, prior load production)
c. How many records would be added to DEV schema table
d. How many active DEV records will be marked for deletion
In most cases number of records in stage (a) should be greater than number of active records in production (b)
In rare occasion that might not be the case. It is responsibility of
the developer running the scripts to verify that that is the the valid.
$ sqlplus READ_20140131/myPass@DEV_VOCAB
Execute the following SQL commands:
SELECT '- Num Rec in stage' AS scr, COUNT (8) cnt FROM SOURCE_TO_CONCEPT_MAP_STAGE c WHERE c.SOURCE_VOCABULARY_ID IN (17) UNION ALL SELECT '- Num Rec in DEV not deleted' AS scr, COUNT (8) cnt FROM DEV.SOURCE_TO_CONCEPT_MAP d WHERE d.SOURCE_VOCABULARY_ID IN (17) AND D.TARGET_VOCABULARY_ID IN (01) AND NVL (d.INVALID_REASON, 'X') <> 'D' UNION ALL SELECT '- How many records would be new in DEV added' AS scr, COUNT (8) cnt FROM SOURCE_TO_CONCEPT_MAP_STAGE c WHERE c.SOURCE_VOCABULARY_ID IN (17) AND c.TARGET_VOCABULARY_ID IN (01) AND NOT EXISTS (SELECT 1 FROM DEV.SOURCE_TO_CONCEPT_MAP d WHERE d.SOURCE_VOCABULARY_ID IN (17) AND c.SOURCE_CODE = D.SOURCE_CODE AND d.SOURCE_VOCABULARY_ID = c.SOURCE_VOCABULARY_ID AND d.MAPPING_TYPE = c.MAPPING_TYPE AND d.TARGET_CONCEPT_ID = c.TARGET_CONCEPT_ID AND d.TARGET_VOCABULARY_ID = c.TARGET_VOCABULARY_ID) UNION ALL SELECT '- How many DEV active will be marked for deletion' AS scr, COUNT (8) cnt FROM DEV.SOURCE_TO_CONCEPT_MAP d WHERE d.SOURCE_VOCABULARY_ID IN (17) AND D.TARGET_VOCABULARY_ID IN (01) AND NVL (d.INVALID_REASON, 'X') <> 'D' AND d.VALID_START_DATE < TO_DATE ( SUBSTR (USER, REGEXP_INSTR (USER, '_[[:digit:]]') + 1, 256), 'YYYYMMDD') AND NOT EXISTS (SELECT 1 FROM SOURCE_TO_CONCEPT_MAP_STAGE c WHERE c.SOURCE_VOCABULARY_ID IN (17) AND c.SOURCE_CODE = D.SOURCE_CODE AND d.SOURCE_VOCABULARY_ID = c.SOURCE_VOCABULARY_ID AND d.MAPPING_TYPE = c.MAPPING_TYPE AND d.TARGET_CONCEPT_ID = c.TARGET_CONCEPT_ID AND d.TARGET_VOCABULARY_ID = c.TARGET_VOCABULARY_ID) AND EXISTS (SELECT 1 FROM SOURCE_TO_CONCEPT_MAP_STAGE c WHERE d.SOURCE_CODE = c.SOURCE_CODE AND d.SOURCE_VOCABULARY_ID = c.SOURCE_VOCABULARY_ID AND d.MAPPING_TYPE = c.MAPPING_TYPE AND d.TARGET_VOCABULARY_ID = c.TARGET_VOCABULARY_ID);
|
Current Result:
- Num Rec in stage 62800
- Num Rec in DEV not deleted 54867
- How many records would be new in DEV added 3893
- How many DEV active will be marked for deletion 312
1.5.3 Load new maps into DEV schema concept table,mark deprecated concepts as deleted. Valid start date, valid end date and invalid
reason are left empty for now.
Transfer records from SOURCE_TO_CONCEPT_MAP_STAGE to DEV.SOURCE_TO_CONCEPT_MAP table
$ sqlplus READ_20140131/myPass@DEV_VOCAB @17_load_maps.sql