[Xitami-Dev] Large Volume File Transfer

[Kimman Balakrishnan]

Hello all

Our application area is document imaging including production environments.

With the growing trend towards offshore processing, we now are faced with customers
scanning on-site and then transferring files to an offshore server. The quantum of
data could be anywhere from 500 MB per day to a few GB. Some of the customers have
dedicated 'pipes' which have to be used. Typically there is a dedicated pipe to a
US HQ and then one or more dedicated pipes to offshore servers. This implies a
multi-step transfer. And to complicate things a little more, there can be time
slots for the use of the pipes for these transfers.

It seems like we could build this using the SMT architecture.

At the originating site, we have a FTP client - this could be built as an SMT
agent.  Happily the FTP RFC includes state diagrams and should make writing the
agent relatively easier.

At all other sites we have a Xitami webserver with the FTP agent running. If the
server is an 'intermediate' site, the server includes the FTP client agent as well.
Once a 'set' of files are recd by the server, a message is sent to the FTP client
agent to initiate the transfer to the next server. Each 'set' is a batch of images
with a control file - the onward sending starts only when the 'set' is complete -
this validation is specific to our context.

The FTP client configuration will include :

a. the alternate routes available for transfer - this would be in the intermediate
sites.
b. the number of simultaneous transfers that can be initiated
c. TCP/IP configuration to ensure max throughput - I have found some articles on
the TCP/IP configuration for these kind of large transfers

Comments and suggestions are invited.

Kimman Balakrishnan
Imaging Solutions Pvt Ltd