Cloud Integration – Poll-Enrich with SFTP Adapter for multiple file polling

This blog describes a way to configure the Poll Enrich pattern in combination with the SFTP adapter to transfer files from SAP S/4HANA to an external SFTP.

Background Information

While working on a SAP S/4HANA implementation project, there was a requirement to transfer files generated on the SAP S/4HANA (AL11) to an external SFTP.
There are some processes that create these files and that need to be transferred to an external SFTP.

Systems and Platforms

SAP S/4HANA FOUNDATION 2020
SAP BTP Integration Suite -> Cloud Integration -> Cloud Foundry 6.30.17
External SFTP – SSH File Transfer Protocol

Business Requirement

Periodically check the SAP S/4HANA folders for new files. When a file is available, it needs to be transferred to the external SFTP server through Cloud Integration and then the file is moved to the archive folder.

High-level process overview

The Challenge with this Integration

With the latest SAP Cloud Integration features, the new Poll Enrich pattern in combination with the SFTP adapter was made available with the main possibilities as mentioned in this blog:

(https://blogs.sap.com/2021/05/12/cloud-integration-use-poll-enrich-with-sftp-adapter/)

– Enrich the message payload with additional data retrieved from a file on the SFTP server
– Poll from a SFTP server triggered by an external trigger, for example triggered via HTTPS call
– Poll from a SFTP server but want to define the configuration in the SFTP adapter dynamically, for example from a partner directory

With this, it is possible to execute a call to an SFTP retrieving a file in the middle of the iFlow, which was previously only possible at the beginning of the iFlow (using the Sender channel).

The challenge begins when Poll-Enrich always polls only one file. If you need to poll multiple files you need to do this Poll-Enrich call in a loop.

However, when you trigger the Poll-Enrich call in a loop, for the first time the file is read (SAP_PollEnrichMessageFound = true); but in the second time, the file is not read  (SAP_PollEnrichMessageFound = false). 

The issue is with the SFTP poll-enrich feature and the post-processing in a loop. 

To avoid this, the recommendation is to implement the loop in one integration flow and from the loop process call another integration flow via ProcessDirect adapter. The second flow then contains the poll-enrich with SFTP, in this way:

Main Flow -> Loop Process Call -> ProcessDirect Adapter -> Second Flow with PollEnrich with SFTP

Solution Implemented 

Two iFLows were created for this solution:

• First iFlow to orchestrate the pooling
• Second iFlow to execute the pooling

First IFlow to orchestrate the pooling

First IFlow to orchestrate the pooling

1. The possibility to trigger the integration scenario on-demand by HTTPS call
2. The possibility to trigger the integration scenario periodically using a Start Timer
3. The main integration process to:

• • Receive the calls (from block 1 or 2) 
  • Set general properties to control the log level
  • Set the SFTP address by external parameters
  • A Groovy script to get the directories to be accessed from a value mapping:
    
  • A router to evaluate if has directories or not
  • A value to start the loop
  • Call the process that executes the loop
  • Get the final payload that was combined for each file transferred
  • Log the final payload (execution log) as an attachment in the message (if configured)

      4. A local integration process to transfer all files containing:

• • Get the directory of the current loop stage
  • A router to check if there is directory or not
  • Set the SFTP information like Environment, Folder Source and Folder Destination
  • Define the SFTP environment based on the environment configured
  • Log the beginning of the process 
  • Set the loop condition to true
  • Call the process inside the loop
  • Log the ending of the process

      5. A local integration process to call the next iFlow that pool the file

• • Set the payload that will be transferred to the next iFlow
  • Request Reply to call the next iFlow 
  • Get the results of the next iFLow
  • A router to evaluate if a file was found or not
  • Log the result of process (If a file was found or not)

Second IFlow to execute the pooling

Second IFlow to execute the pooling

1. The main integration process to:

• • Explore the payload received from the previously iFlow with the SFTP informations
  • A Poll Enrich to fetch the file
  • A SFTP Sender adapter to fetch the file (any file *)
  • A router to evaluate if a file was found or not
  • If found, send to the External SFTP
  • Fill the body with the results of the execution

Execution log

At the end of the process, an execution log will be presented, as following:

Execution log

Benefits/Advantages of the implemented solution

We can find some benefits and advantages, such as:

• • The file transfer will be automatic, not depending on a person for this
  • An automatic run can be scheduled periodically or also a manual run on demand
  • Multiple directories
  • All files in the directory will be transferred 
  • No need for SAP S/4HANA to communicate directly with the external SFTP
  • The integration scenario is in the Cloud Integration not influencing the functioning of SAP S/4HANA

Possible improvements of the implemented solution

We can find some possible improvements, such as:

• • Transfer files to more than one external SFTP
  • Add exception routines in case of errors

Conclusion

With this solution, it is possible to transfer files from SAP S/4HANA to an external SFTP in a configurable and robust way.

I would be more than happy to hear from our SAP Community about the similar experiences with this requirement.
Please share here in the comments your feedback, thoughts and if you have similar requirements, how you implemented them.

Thank you for reading!!

Fellipe Mendes