Data Ingestion
This section provides the details on Ingester functions, Data flow, Initial load and field mapping in the Marketing Catalog microservice.
Marketing Catalog microservice receives the product update information as events asynchronously from the record-keeping system (for example, Temenos Transact). Using the Product-ID, Marketing Catalog microservice in turn calls an IRIS API to fetch all the details of the product and then it updates the microservice database.
Currently, Marketing Catalog provides three Ingester functions which help to ingest Temenos Transact product events.
- TransactProductLineIngester — Inserts the Product Line data into the ms_marketingcatalog_productline table in the Marketing Catalog microservice database.
- TransactProductGroupIngester — Inserts the Product Group data into the ms_marketingcatalog_productgroup table in the Marketing Catalog microservice database.
- TransactProductIngester — Inserts the Product data into the ms_marketingcatalog_productinformation table in the Marketing Catalog microservice database.
- TransactPeriodicInterestIngester — Inserts the Periodic Interest data in to ms_marketingcatalog_periodicinterest table in Marketing Catalog Microservice database.
- TransactBasicInterestIngester — Inserts the Basic Interest data in to ms_marketingcatalog_basicinterest table in Marketing Catalog Microservice database.
Ingester listens to events on the streaming platform configured (Kafka) and ingests data asynchronously in near real-time in the Marketing Catalog microservice database. The underlying streaming platform does not provide a message sequence guarantee to the microservice application.
Data Flow
Initial load processing triggers events for existing business data related to Marketing Catalog in Temenos Transact. The data flow follows the following route:
T24 Application tables → F_DATA_EVENTS → DES (Kafka) → MS Marketing-Catalog Ingester → IRIS API Call → MS Marketing-Catalog Ingester → MS Marketing-Catalog DB
Where,
- Temenos Transact uses Relational Replication to load the application records (AA.PRODUCT) to F_DATA_EVENTS table.
- DES (Data Event Stream) fetches and transforms the record from F_DATA_EVENTS to Kafka topics.
- In microservice, ingestor consumes kafka topic and store it in the underlying microservice database.
The following are the Temenos Transact applications:
- AA.PRODUCT
- AA.PRODUCT.LINE
- AA.PRODUCT.GROUP
- BASIC.INTEREST
- PERIODIC.INTEREST
Processing Initial Load
This section helps you to perform the initial load on Marketing Catalog microservices database tables with Temenos Transact application records.
The prerequisite before you perform initial load is to run the following:
- Marketing Catalog microservice - Ingester topic should be pre-configured (See Deployment Instructions section for more information).
- Data Event Stream (DES)
- Temenos Transact
- IRIS end point for Release version 202010 or before - v1.0.0/product/HOME.EQUITY.LOAN/productDetails
- IRIS end point from release version 202101 - v1.0.0/product/HOME.EQUITY.LOAN/productConditions)
The Ingestion should happen sequentially in the following order:
- BASIC.INTEREST
- PERIODIC.INTEREST
- AA.PRODUCT.LINE
- AA.PRODUCT.GROUP
- AA.PRODUCT
Procedure:
- Create a record in RR.PARAM application for FXXX.BASIC.INTEREST, FXXX.PERIODIC.INTEREST, FXXX.AA.PRODUCT, F.AA.PRODUCT.LINE, and FXXX.AA.PRODUCT.GROUP application with ILP.REQUIRED set to YES.
- After the successful commit, the schema registry in DES will be updated with BASIC_INTEREST, PERIODIC_INTEREST, AA_PRODUCT, AA_PRODUCT_LINE and AA_PRODUCT_GROUP schemas
Schema Registry - localhost:8081/subjects
["DesInstallEvent","CATEG_ENTRY","SECTOR","BASIC_INTEREST","PERIODIC_INTEREST", "DataEventIdEot","AA_PRODUCT","AA_PRODUCT_LINE","AA_PRODUCT_GROUP","DataEvent","STMT_ENTRY","RE_CONSOL_SPEC_ENTRY","SchemaInfoEvent","DataEventIdRecovery"]
- Start the Service TSM and BNK/RR.TAKEON in Transact.
- Launch the TSM tRun START.TSM –DEBUG.
- Launch the agent allocated for the service BNK/RR.TAKEON using the commane tRun tSA <AgentNumber>.
- All the BASIC.INTEREST, PERIODIC.INTEREST, PRODUCT.LINE, PRODUCT.GROUP and published AA.PRODUCT records are loaded into the marketing catalog micro service after the completion of the service.
Field Mapping
The field mapping from Temenos Transact to Marketing Catalog microservice is shown in the following table.
|
MS Entity |
MS Attribute |
Sub Attribute |
Data Type |
Description |
Transact Application |
Transact Field Name |
|---|---|---|---|---|---|---|
|
Product Line |
productLineRef |
String |
The top-level product line in which the product is ultimately attached |
AA.PRODUCT.LINE |
REC.ID |
|
|
productLineName |
String |
Specifies the display name of the Product Line |
AA.PRODUCT.LINE |
DESCRIPTION |
||
|
externalIndicator |
boolean |
Indicates the external product line |
AA.PRODUCT.LINE |
REC.ID |
||
|
Product Group |
productGroupRef |
String |
The groups under a specific product line in which the product is ultimately grouped. |
AA.PRODUCT.GROUP |
REC.ID |
|
|
branchRef |
String |
Unique identifier of the branch |
AA.PRODUCT.GROUP |
CO.CODE |
||
|
productLineRef |
String |
The top-level product line in which the product is ultimately attached |
AA.PRODUCT.GROUP |
PRODUCT.LINE |
||
|
externalIndicator |
boolean |
Indicates the external product group |
AA.PRODUCT.GROUP |
GROUP.TYPE |
||
|
Product Information |
productRef |
String |
Unique identifier for the product in the core banking system |
AA.PRODUCT |
REC.ID |
|
|
productName |
String |
Specifies the display name of the Product |
AA.PRODUCT.DESIGNER |
DESCRIPTION |
||
|
productLine |
String |
The top-level product line in which the product is ultimately attached |
PRODUCT.GROUP |
PRODUCT.LINE |
||
|
productGroup |
String |
The groups under a specific product line in which the product is ultimately grouped. |
AA.PRODUCT.DESIGNER |
PRODUCT.GROUP |
||
|
branchRef |
String |
Unique identifier of the branch |
AA.PRODUCT.DESIGNER |
CO.CODE |
||
|
availableFrom |
date |
Date from when the product is available. |
AA.PRODUCT |
CAT.AVAILABLE.DATE |
||
|
availableTo |
date |
Date until when the product is available. |
AA.PRODUCT |
CAT.EXPIRY.DATE |
||
|
externalIndicator |
boolean |
Indicates the external products |
PRODUCT.GROUP |
PRODUCT.LINE |
||
|
productFeatures |
type |
String |
Specifies the type of the product features |
AA.PRD.CAT.FACILITY |
SERVICE |
|
|
isMandatory |
Boolean |
Specifies whether the product feature is mandatory or not |
AA.PRD.CAT.FACILITY |
SERVICE.AVAILABILITY |
||
|
defaultValue |
String |
Specifies the customer option for the product features |
AA.PRD.CAT.FACILITY |
CUSTOMER.OPTION |
||
|
optionValue |
value |
String |
Specifies the available customer option for the product features |
STANDARD.SELECTION |
SYS.VAL.PROG |
|
|
Product Details |
currency |
String |
Currency of the product |
AA.PRODUCT.DESIGNER |
CURRENCY |
|
|
Product Conditions
|
currency |
String |
Currency of the product |
AA.PRODUCT.DESIGNER |
CURRENCY |
|
|
InterestDetails |
property |
String |
Indicates the reference for Interest property condition |
AA.PRD.CAT.INTEREST |
REC.ID |
|
|
period |
String |
Indicates the property conditions applicable for the defined period |
AA.PRD.CAT.INTEREST |
REC.ID |
||
|
effectiveDate |
date |
This field represents the effective date from which these product conditions come into force. |
AA.PRODUCT.DESIGNER |
EFFECTIVE.DATE |
||
|
rateTierType |
String |
Each tier is specified by defining the amount up to which the interest rate applies. Additionally, each tier can be of a different interest rate type. |
AA.PRD.CAT.INTEREST |
RATE.TIER.TYPE |
||
|
dayBasis |
String |
Type of day basis used for interest calculation |
AA.PRD.CAT.INTEREST |
DAY.BASIS |
||
|
compoundType |
String |
Compounding interest applies a factor to the compound interest calculation based upon the number of compounding periods in the year. This field defines the number of compounding periods to be applied. |
AA.PRD.CAT.INTEREST |
COMPOUND.TYPE |
||
|
compoundYield |
String |
This field indicates how the effective rate is calculated, when set to YIELD the interest rate defined is taken as the annual effective rate |
AA.PRD.CAT.INTEREST |
COMPOUND.YIELD.METHOD |
||
|
interestDetails. creditInterests / interestDetails. debitInterests |
tierAmount |
decimal |
The tier amount related to the field RATE.TIER.TYPE |
AA.PRD.CAT.INTEREST |
TIER.AMOUNT |
|
|
fixedRate |
decimal |
The fixed interest rate used to calculate the loan for the defined period. |
AA.PRD.CAT.INTEREST |
FIXED.RATE |
||
|
floatingIndex |
String |
This field defines the basic interest key that needs to be used for calculating the interest based on floating interest |
AA.PRD.CAT.INTEREST |
FLOATING.INDEX |
||
|
periodicIndex |
String |
This field defines a periodic rate is tied to an index (e.g. LIBOR) which is dependent on a period of time (e.g. term) and possibly an amount |
AA.PRD.CAT.INTEREST |
PERIODIC.INDEX |
||
|
periodicPeriod |
String |
Defines the interest period to be used when linking to PERIODIC.INTEREST. |
AA.PRD.CAT.INTEREST |
PERIODIC.PERIOD |
||
|
periodicMethod |
String |
If system does not find a interest rate in the correct bucket of PERIODIC.INTEREST table, it automatically applies a method to arrive at a rate chosen by this method. |
AA.PRD.CAT.INTEREST |
PERIODIC.METHOD |
||
|
periodicReset |
String |
This indicates the reset period of the index stated. |
AA.PRD.CAT.INTEREST |
PERIODIC.RESET |
||
|
creditInterests. marginDetails / debitInterests. marginDetails |
type |
String |
Identifies whether there are single or multiple margins |
AA.PRD.CAT.INTEREST |
MARGIN.TYPE |
|
|
marginOperand |
String |
The options are ADD or SUB. This indicates whether the MARGIN.RATE should be added or subtracted from the interest rate. |
AA.PRD.CAT.INTEREST |
MARGIN.OPER |
||
|
marginRate |
decimal |
A margin rate can be used to adjust the specified rate of interest and to appropriately reflect any rate profit realized. The result is the nominal rate of interest. |
AA.PRD.CAT.INTEREST |
MARGIN.RATE |
||
|
ChargeDetails |
property |
String |
Indicates the reference for Charge property condition |
AA.PRD.CAT.CHARGE |
REC.ID |
|
|
chargeCurrency |
String |
Defines the currency of the charge property. Charges will be calculated for the Currency defined in this field. |
AA.PRD.CAT.CHARGE |
CURRENCY |
||
|
period |
String |
Indicates the property conditions applicable for the defined period |
AA.PRD.CAT.CHARGE |
REC.ID |
||
|
effectiveDate |
date |
This field represents the effective date from which these product conditions come into force. |
AA.PRODUCT.DESIGNER |
EFFECTIVE.DATE |
||
|
type |
String |
Specifies whether the charge is Adhoc or Scheduled |
AA.PRD.CAT.ACTIVITY.CHARGE |
CHARGE |
||
|
chargeType |
String |
This field defines if the charge amount is fixed amount or to be calculated. |
AA.PRD.CAT.CHARGE |
CHARGE.TYPE |
||
|
fixedAmount |
decimal |
This defines fixed charge amount to be returned. |
AA.PRD.CAT.CHARGE |
FIXED.AMOUNT |
||
|
PaymentSchedules |
property |
String |
Indicates the reference for schedules property condition |
AA.PRD.CAT.PAYMENT.SCHEDULE |
REC.ID |
|
|
period |
String |
Indicates the property conditions applicable for the defined period |
AA.PRD.CAT.PAYMENT.SCHEDULE |
REC.ID |
||
|
effectiveDate |
date |
This field represents the effective date from which these product conditions come into force. |
AA.PRODUCT.DESIGNER |
EFFECTIVE.DATE |
||
|
paymentSchedules. paymentTypes |
paymentType |
String |
Indicates the type of the payment |
AA.PRD.CAT.PAYMENT.SCHEDULE |
PAYMENT.TYPE |
|
|
method |
String |
Payment method by which the repayments are made. |
AA.PRD.CAT.PAYMENT.SCHEDULE |
PAYMENT.METHOD |
||
|
payFrequency |
String |
This field indicates the frequency at which the payments will be made due. |
AA.PRD.CAT.PAYMENT.SCHEDULE |
PAYMENT.FREQ |
||
|
properties |
Collection<String> |
The interest and charges property for which the schedules are applicable. |
AA.PRD.CAT.PAYMENT.SCHEDULE |
PROPERTY |
||
|
PaymentTypes. paymentDates |
startDate |
String |
This field indicates the actual payment start date. |
AA.PRD.CAT.PAYMENT.SCHEDULE |
START.DATE |
|
|
endDate |
String |
An End Date can be specified to indicate when the payments should terminate |
AA.PRD.CAT.PAYMENT.SCHEDULE |
END.DATE |
||
|
TermAmounts |
property |
String |
Indicates the reference for Term amount property condition |
AA.PRD.CAT.TERM.AMOUNT |
REC.ID |
|
|
period |
String |
Indicates the property conditions applicable for the defined period |
AA.PRD.CAT.TERM.AMOUNT |
REC.ID |
||
|
effectiveDate |
date |
This field represents the effective date from which these product conditions come into force. |
AA.PRODUCT.DESIGNER |
EFFECTIVE.DATE |
||
|
amount |
decimal |
Defines the total amount which will be lent or deposited for the term (i.e. the committed amount). |
AA.PRD.CAT.TERM.AMOUNT |
TERM |
||
|
term |
String |
Determines the period of time by which the amount must be repaid (either to the bank in the case of a loan or to the customer in the case of a deposit) During product definition it will be common to default the term (e.g. a 30 year mortgage) |
AA.PRD.CAT.TERM.AMOUNT |
AMOUNT |
||
|
ReportingDetails |
property |
String |
Indicates the reference for Reporting property condition |
AA.PRD.CAT.REPORTING |
REC.ID |
|
|
effectiveDate |
date |
This field represents the effective date from which these product conditions come into force. |
AA.PRODUCT.DESIGNER |
EFFECTIVE.DATE |
||
|
reportingDetails. aprTypes |
aprType |
String |
Indicates the reference for APR calculation. |
AA.PRD.CAT.REPORTING |
APR.TYPE |
|
|
excludeProperty |
Collection<String> |
Indicates properties which will be excluded for APR calculation for the given APR scenario. |
AA.PRD.CAT.REPORTING |
EXCLUDE.PROPERTY |
In this topic
