Salaryman IT Ltd - Infrastructure and Database Migrations

Oracle and Database Migration Links

Database Migration Links

File Transfer Accelerators/File Copy Hints and Tips:

Migrating databases means there will be a requirement for copying database files or export dumps from one location to another. Typically, network speed between the two locations will be configured for normal business workloads, not for the huge database file lifts that migration projects can require. We therefore may need to identify ways to make the most of the network capacity we have. In very extreme cases it may require copying files to tapes and physically transferring them to the target location, but usually we can get the performance we need either by multithreading our file copy jobs using standard file copy utilities, or by adopting a File Transfer Accelerator (FTA) utility.

The need for FTAs arises because standard TCP-based utilities such as sftp, ssh, scp or whatever don't work efficiently. Processes can end up spending more time waiting for acknowledgements than sending data due to the way they work - TCP is serial in nature and it wont send the next set of data until the previous one is completed, including resolution of lost packets, hence the wait time. The FTAs that are available are based on the UDP protocol, rather than TCP. UDP isnt as robust as TCP in terms of managing packet loss, so the point of the FTAs is to provide that missing functionality in a way that avoids the latency issues that TCP experiences. The FTAs also include appropriate Security and other functionality to provide a complete and robust file transfer solution.

In an otherwise unloaded network link I have witnessed transfer rates at very close to the theoretical line speed, thus maximising the throughput that is possible for a given line. For example, for a 100 mbps line (45 GB/h) I have seen transfer rates above 40 GB/h using FTA utilities, compared to anything from maybe 2-12 GB/h using standard TCP-based utilities.

FTAs I have encountered include these...

Aspera : I have used Aspera File Transfer Accelerator tools on two projects. They are cheap, flexible, easy to install, easy to use and achieve considerable improvements of data throughput across networks compared to standard TCP-based utilities.

FileCatalyst : FileCatalyst sounds similar to Aspera, but I havent used it to date so have no experience to base any opinion on. The linked page on the FileCatalyst website does, however, provide a useful summary of why using ftp and TCP can be slow.

Linux Tip: Super-Fast Network File Copy : Some tips on using standard Linux utilities like netcat and ssh to optimise throughput of file copies

Tsunami UDP - Tsunami UDP is an open source file transfer protocol which is suggested by Amazon Web Services to use when transferring files to the AWS Cloud. I havent had an opportunity to test it, but it looks similar to Aspera and FileCatalyst, so presumably offers similar functionality and performance to tools those, but with a lower price tag!

Oracle Datapump Links:

Database Migration Resources:

EMC - Planning a Data Centre Migration - Good article covering some of the key success factors for relocating a large data center to a new facility. - Documentation for Data Centre Migrations

SearchITChannel - Three steps to a successful data center migration plan

InfoSys - Successful Data Center Migration - eBook covering an end to end approach for Data Center migrations

Migrating to the Cloud - Technet Article on The title refers to the Cloud, but the article itself is pretty reticent about it and the theme is really "stuff to consider when migrating from non-Oracle to Oracle". Nevertheless, it offers a pretty good high-level summary of how to undertake such migrations.

Amazon RDS - Amazon Relational Database Service (Amazon RDS) is a web service that makes it easy to set up, operate and scale a relational database in the cloud. This link describes the technical options currently available to migrate to Amazon RDS.

Data Migration and ETL Tool Links

Migration to Oracle from Non-Oracle

Oracle Migration Path - Oracle Partner Network Article showing the recommended database migration path to follow when using the services available from the Oracle Platform Migrations Group (PMG). Includes numerous links to other resources and information sources, for example...

ISV - Migrating to Oracle 9i/10g - Oracle slide deck talking about migration from non-Oracle to Oracle and the Oracel Migration Workbench

Third-Party Migration Providers - Oracle's preferred third-party migration providers offering consulting in a variety of locations for a fixed price or on a time and materials basis

Copyright 2013 David Orchard Consulting Limited. All rights reserved. Tel +44 (0) 7753249705,