Google

21 November 2013

Moving Millions Of Files Into SharePoint - An Interesting Challenge

I have recently been hired to lead a project where millions of files will be moved into SharePoint. It is not an easy task, because there are many things to consider, but it is an interesting challenge that I am happy to take on.

Consultation assignments
The first kalmstrom.com solutions have been with me for nearly fifteen years, and the kalmstrom.com brand is the business creation I am most proud over. But I have a team of skilled co-workers assisting me with the kalmstrom.com development and documentation, so I have time to accept assingments outside my own company.

These assignments usually gives me interesting input from new people and challenges that make me grow as a developer. I can often make use of the knowledge and inspiration I get from these assignment when I design new versions of my own products.

SharePoint 2013
SharePoint 2013 logoI am a Microsoft certified SharePoint expert, and it was my SharePoint knowledge and experience that qualified me for this assignment. I have now set up a prototype of a SharePoint 2013 farm that will be used by the IT department as a first prototype.

2 million files in 225 000 folders 
The organization needs to keep a huge amount of documents on file in a way that is both secure and easy to search. It has 225 000 directories containing around 2 million files, so my first task was to do a file server inventory.

The inventory has given an opportunity to clean-up and determine what should be kept for the future. It is now completed, and we can go over to the difficult decisions on what to do with all the files.

PowerShell automates tedious work
PowerShell iconBefore moving the 225 000 folders to SharePoint, we must consider what should become of them. Should the content of the folder be transformed into a site collections, a site or a document library, or should it be used as metadata or even tags? Is the information stored in Excel lists suitable to export into SharePoint lists?

To migrate the data manually would take years, but once the decisions are made the actual migration is done with PowerShell. All the SharePoint containers are automatically created and populated with the correct files.

Metadata should be kept
An important aspect to consider is the existing metadata, that should not be lost in the transfer to SharePoint. Such metadata could be the information already tagged by the file location. A file that exists under P:\Projects\2010\IT\ClientPlatform\ already has a lot of metadata that should be maintained. It should be tagged as a project file, created in 2010, by the IT-department, concerning the ClientPlatform project.

Office update
Office 2013 logo In the process I am engaged in, the organization also converts all of their Office files into the new file formats. That means that the old .doc and .xls files will be converted into .docx and .xlsx formats, which gives a huge space saving benefit and enables the new Office features.

Search options
Another consideration when I design the new SharePoint system is how information should be found in the future. It should still be possible to find information by location, but we should also use the SharePoint search features. Here I can do a lot by using the SharePoint features in combination with the maintained metadata.

Long term project
This SharePoint project will probably continue a for at least six months more. It would be difficult for me to work full time with a big project for such a long time, but the managers are very flexible and I can devote myself to this project when I am free for other commitments. This is a situation that suits me very well, and I am looking forward to seeing the project develop. .
By Peter Kalmstrom
CEO and Systems Designer
kalmstrom.com Business Solutions