O365 HIGH SPEED MIGRATION TECHNICAL SOLUTION
Background
Migration of
huge volumes of content from on-premises installations of SharePoint to multi-tenant
O365 SharePoint Online used to be very time consuming using traditional CSOM
APIs provisioned by Microsoft.
Migration of
content volumes exceeding 500 GB which is heavily dependent on network factors
like bandwidth, availability etc., will run into months.
Moreover,
Microsoft imposes a number of restrictions on migrations using the traditional
CSOM path.
Challenges using Traditional CSOM for Content Migration
·
Limitations on the number of
Client Side Object Model (CSOM) calls, a particular user can initiate on Office
365 within a given timeframe – referred as user based throttling.
·
Limitations on the number of
CSOM calls that can be initiated against a particular Office 365 tenant or
SharePoint Online farm within a given timeframe – referred as tenant based throttling.
·
Bandwidth limitations when
copying content from on-prem to O365.
Note: Depending on usage, Microsoft used to fine-tune
thresholds so that users can consume the maximum number of resources without
degrading the reliability and performance of SharePoint Online. Hence no
strictly defined thresholds/limitations are shared.
In order to
address the problem of large-scale migrations to O365 SharePoint online (that
can potentially have adverse cost and timeline implications), Microsoft has
launched a set of High Speed Migration APIs which can significantly boost
migration speed by leveraging Microsoft Azure.
Approach
Microsoft provides
four-steps approach using PowerShell scripts.
System Requirement
Supported Operating Systems:
Windows 7 Service Pack 1, Windows 8, Windows Server 2008 R2 SP1,
Windows Server 2008 Service Pack 2, Windows Server 2012, Windows Server 2012 R2
Software:
PowerShell 4.0
Prerequisites
·
An Azure subscription. Azure Subscriptions
·
An Azure Storage account.
·
Provision your Office 365
§
with your existing
active directory Office 365
integration with on-premises environments
OR
OR
§ Use one of the other options for adding accounts to Office 365 Add users to
Office 365 for business.
·
Download and install the SharePoint
Online Management Shell MSI. Use the
Control Pane to uninstall any previous versions.
STEP 1 – Create Target Site Structure
Site
structure on the
target environment including
Sites/Subsites/Lists/Libraries etc is created
via Migration Tool/CSOM/PowerShell.
STEP 2 – Create packages
Source
Content is
packaged via PowerShell. Steps are as follows.
A.
Install SharePoint Online Management shell
Note: Uninstall all previous versions. Open the SharePoint Online Management Shell
as an Administrator.
B.
Set up your working directory
Create two
empty folders before you start the migration process. These folders do not require a lot of disk
space as they will contain only XML.
·
Create a Temporary package
folder
·
Create a Final package folder
C.
Create an Azure Storage Account
Keep Azure
Storage account name and Key ready.
D.
Determine your locations and credentials
$sourceFiles =
'\\fileshare\users\charles'
$sourcePackage =
'C:\migration\CharlesDocumentsPackage_source'
$targetPackage =
'C:\migration\CharlesDocumentsPackage_target'
$targetWeb =
'https://contoso-my.sharepoint.com/personal/charles_contoso_com'
$targetDocLib =
'Documents’
$azureAccountName =
'contosomigration’
$azureAccountKey =
'Insert Azure Account Key string Here’
$azureQueueName =
'migrationqueue’
E.
Create new content package
After you
determine your locations and credentials, the next step is to create a new
migration package from a file share. The New-SPOMigrationPackage
command reads the list of content targeted by the source path and will generate
XML to perform migration.
There are two required parameters to enter (others are optional)
· SourcefilesPath:
points to the content you want to migrate
·
OutputPackagePath: points to your Temporary folder
Sample Syntax:
New-SPOMigrationPackage -SourceFilesPath $sourceFiles
-OutputPackagePath $sourcePackage
F.
Convert the content package for your target site
After you have
created the content package, the ConvertTo-SPOMigrationTargetedPackage
command converts the xml generated in your temporary folder. It saves a new set
of targeted migration package metadata files to the target directory. This is the final package.
Note: Your target site collection administrator
credentials are used to gather data to connect to the data site
collection.
There are six
required parameters to enter (others are optional)
· TargetwebURL: points
to the destination Web
· SourceFilesPath: points
to the you want to migrate
· SourcePackagePath: points
to your Temporary package folder
· OutputPackagePath: points
to your final package folder
· TargetDocumentLibraryPath: the path to your destination library
· Credentials: SPO
credential that has admin rights to the destination site
Sample Syntax:
# Convert package to a targeted one by looking up data in
target site collection
ConvertTo-SPOMigrationTargetedPackage
-SourceFilesPath $sourceFiles -SourcePackagePath $sourcePackage
-OutputPackagePath $targetPackage -TargetWebUrl $targetWeb
-TargetDocumentLibraryPath $targetDocLib -Credentials
STEP 3 – Upload Package
In this step, migration package is created
in containers inside Azure storage using your storage account credentials using
the Set-SPOMigrationPackageAzureSource
command. The package files are uploaded to the container, a snapshot of the
files completed, and then it returns the connection strings to a PowerShell
variable.
There are five required parameters (others
are optional)
· SourceFilePath: points to the content you want to migrate
· SourcePackagePath: points to your final package folder
· AccountName: the name of the Azure storage account
· Accountkey: the key to the Azure storage account
· AzureQueueName: The name of the Azure queue BLOB (optional
but recommended)
Recommendation: Take the output of the following command and
store it in a variable for the next command.
$temporaryVariable =
Set-SPOMigrationPackageAzureSource
Sample Syntax:
# Create azure containers and upload package into them,
finally snapshotting all files
$al =
Set-SPOMigrationPackageAzureSource -SourceFilesPath $sourceFiles
-SourcePackagePath $targetPackage -AzureQueueName $azureQueueName -AccountName
$azureAccountName -AccountKey $azureAccountKey
$al|fl
STEP 4 – Submit to Migrate to O365
The final step is to use the Submit-SPOMigrationJob command to
create a new migration job in the target site
collection, and returns a GUID representing the JobID.
There are three
required parameters (others are optional)
· TargetwebURL: point
to the Web of the destination
· MigrationPackageAzureLocations: the temporary variable from the previous command
· Credentials: SPO
credential that has admin rights to the destination site
Sample Syntax:
# Submit package data to site
collection to create new migration job
Submit-SPOMigrationJob
-TargetWebUrl $targetWeb -MigrationPackageAzureLocations $al -Credentials
$creds
Job Status can be checked in the Azure
storage account queue. Real time updates will be posted there.
Manifest container in the Azure Storage can
be checked for logs of everything that happened. It is now safe to delete those containers once
migration is completed.
If there were errors or warnings, .err and
.wrn files will be created in the manifest container.