Skip to content

smartbiz/talend

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Talend Developer Guide

This Developer Guide will introduce you how to use Talend Open Studio for ESB to develop Data Integration, develop and deploy Enterprise Service Bus and finally manage and monitor the deployed services. #Table of Contents# 1. [Eviroment Setup](#TOS-evn-01) 1. [Data Integration](#TOS-Job-01)
2.1. [Job Data Flow](#TOS-Job-02)
2.2. [Write your own Job](#TOS-Job-03)
2.3. [Execute and Debug](#TOS-Job-04) 2. [Enterprise Service Bus](#TOC-ESB1)
3.1. [Routes](#TOC-ESB2)
3.1.1. [ActiveMQ Integration](#TOC-ESB3)
3.2. [Service Development](#TOC-ESB4)
3.2.1. [SOAP CRM service](#TOC-ESB5)
3.2.2. [REST Customers service](#TOC-ESB6)
3.3. [Service deployment](#TOC-ESB7)
3.3.1 [How to deploy service using ESBRuntime Container](#TOC-ESB8)
3.3.2. [deploy service using apache tomcat](#TOC-ESB9) 3. [Management and Monitoring](#TOC-ESB10)
4.1.1. [Hawtio](#TOC-ESB11)

EnvironmentSetup

  • Download Java SE Development Kit 8 from Oracle Website.
  • Install the downloaded package into local directory. The installation path must NOT contain white space.
  • Configure path for JAVA_HOME.

Talend Package Installation

  • Download Talend Open Studio for ESB here
  • Extract the downloaded package into your local directory. Ex: E:\Talend\01.DevTools
  • Go into folder Studio [Ex: E:\Talend\01.DevTools\Studio] and run [TOS_ESB-win-x86_64.exe] in case of Win64bit or [TOS_ESB-win32-x86] in case of Win32bit to start Talend Open Studio for ESB.

Initial View of Talend Open Studio

*Initial View of Talend Open Studio for ESB* # Data Integration # In this part, we will use Initial Data Load module to introduce how to implement Data Integration using Talend Open Studio. ## Job Data Flow ## Talend Open Studio allow you to develop job to implement Data Integration. The purpose of a job is to parse an input file (ex: csv file, xml file, excel file...), then perform validate the raw data, filter the invalid data and finally send the valid data to the server. In order to understand the flow clearly, we will analyze the existing project INITIAL\_DATA\_LOAD. ### Import existing project ### At the initial screen of Talend Open Studio, select option **Import an existing project**, click **Finish** and then browse to the INITIAL\_DATA\_LOAD] project (Please extract [INITIAL\_DATA\_LOAD.zip] in directory ../sources).

Import Project View

After import successfully, the development view will be displayed as below:

IDE view

Understand Job Data Flow

On the Repository, open ProductJob 0.1. The design of ProjectJob 0.1 will be displayed.

Design of a Job

  1. parseInputCSVfile: this component parses the input csv file.
  2. Default value and Validation: At this component, each csv record will be validated the data definition such as data length, data format, data type.etc... The invalid data will be marked for filtering in the next step. Default value is also assigned to the record at this step.
  3. Filter flag: At this step, the valid data will be transferred to the Reference check step. The invalid data will be moved to Log error component.
  4. References check: This step makes sure the input data having the valid reference. For example, when importing a product, the price list must be existed. The data having invalid reference will be marked.
  5. Filter data: The data with invalid reference will go to step Log error. The valid data will go to the final step Finish.
  6. Finish: At this component, all the valid data will be send to ERP server for importing purpose.
  7. Log Error: this component at step 7 and step 8 will log the invalid data.

Write your own Job

This part will guide you steps to write your own simple job. To implement the job, you need the following palette tFileInputDelimited, tJavaRow, tFilterRow.

Step 1: At the Repository, Right Click on Job Design -> Create Job. Input the information for the new Job. Example: name = ProductSampleJob.

Step 2: Using the palette to draw the design as the following image.

Job Creation

Step 3: Select the tFileInputDelimited on the design, then select the Component tab: At the File name/Stream combobox, browse to the csv file. Modify CSV Row Separator, File Separator as in the above image.
Step 4: Select item rawData(Main), at the tab Component, click Edit Schema.

Sample Product Job Schema Selection

Next, click Import schema -> OK

![Image 7](../assets/images/TOS-Job-06.jpg "Import schema") **Step 5**: Select tJavaRow **CheckDefinition**, at the tab **Component**, click **Edit Schema** -> click icon *Copy all of the columns from the input schema to the output schema* -> **OK**. Input the validation code to the **Code** textbox.

Edit schema

Edit Schema -> click icon Copy all of the columns from the input schema to the output schema -> OK.

Copy schema

Step 6: Slect tfilterRow filterFlag, at the tab Component input the Condition to filter at the table Condition

Filter component

Step 7: Write your code to log invalid data into the code textbox of the Component of the component LogRejectProduct and write your code to process valid data into the code textbox of the Component tab of component LogvalidatedProducts.

Code to log result

Job implementation

Execution and Debug

Run

On the tab Run (Job ProductSampleJob), click Run to execute the job.

Run job

Debug

Right click on the component that you want to debug, select Add breakpoint. On the tab ** Run (Job ProductSampleJob)**, select sub-tab Debug Run. On the dropbox Debug:

  • Select Trace Debug if you want to run Debug in Trace mode.

Trace Debug

  • Select Java Debug if you want to run in Java debug mode.

Java Debug

Enterprise Service Bus

Routes

ActiveMQ Integration

Step 1: Start ActiveMQ server

TOS-ESB-16

Step 2: Run MQMultiReceiver job and MQMultiSender in Talend TOS-ESB-17

The result id outputted on the console. TOS-ESB-18

Service Development

SOAP CRM Service###

Step 1: Create service using talend open studio TOS-ESB-19

Step 2: Create Job to expose SOAP

TOS-ESB-20

Step 3: Run job by click run button on Run tab

TOS-ESB-21

Step 4: Test SOAP using boomerang Test case 1: type http://localhost:8090/services/CRMServiceProvider in boomerang and enter id equals 1.

TOS-ESB-22

Test case 2: type http://localhost:8090/services/CRMServiceProvider in boomerang and enter id equals 2.

TOS-ESB-23

Test case 3: Start getCRMStatus function by click run button on run tab and type http://localhost:8090/services/CRMServiceProvider in boomerang and enter id equals 1.

TOS-ESB-24

Test case 4: Start getCRMStatus function by click run button on run tab and type http://localhost:8090/services/CRMServiceProvider in boomerang and enter id equals 2.

TOS-ESB-25

Test case 5: Start updateCRMStatus function by click run button on run tab and type http://localhost:8090/services/CRMServiceProvider in boomerang and update id equals 1.

TOS-ESB-26

The result is outputted on the console.

REST Customer service###

The REST API will be implemented in talend as the following

TOS-ESB-9

Step 1: Create data to runing REST API. select tFixedFieldInput_1->create data in component tab.

TOS-ESB-10

Step 2: Create request for REST API. select tRESTRequest_1->create request in component tab

TOS-ESB-11

Step 3: The Test is implemented on talend (Select Run(job DemoREST)-> Run) and Advanced REST client for chrome.
TOS-ESB-12

Test case 1: type http://localhost:8090/services/customers in Advanced REST client for chrome. TOS-ESB-13

Test case 2: type http://localhost:8090/services/customers/1 in Advanced REST client for chrome.

TOS-ESB-14

Test case 3: type http://localhost:8090/services/customers/2 in Advanced REST client for chrome.

TOS-ESB-15

Service deployment

How to deploy service using ESBRuntime Container

We are ready to deploy Talend ESB RESTful Service DemoREST-0.1 job to Talend Runtime environment for that we must build job from studio as OSGI Bundle for ESB TOS-ESB-2 Select archive file path, Build type as OSGI Bundle For ESB and click Finish

TOS-ESB-3

exported file must be copied in Runtime_ESBSE/container/deploy folder

TOS-ESB-4

On Runtime container use list command to check status of your service.

TOS-ESB-5

Now we are able to deploy ESB job as service in Talend Runtime container using apache karaf but to make it run as a operating system service we must install Karaf Wrapper feature.
Installing the wrapper
Step 1: Browse to the container/bin folder of the Talend Runtime installation directory, then launch the container by executing the trun file as a root user.
Step 2: To install the wrapper feature, simply type:

TOS-ESB-6

Once installed, wrapper feature will provide wrapper:install new command in the trun, which allows you to install Talend Runtime as a service. Step 3: To register the container as a service in automatic start mode, simply type:

TOS-ESB-7

Now, We will test service from chrome brower. Step 1: To test the REST service, type http://10.10.10.1:8040/services/customers (you can replace 10.10.10.1:8040 by your deploy machine) in chrome brower.

TOS-ESB-8

Deploy service using apache tomcat (TODO)

Management and Monitoring (TODO)

Hawtio (TODO)