VSTS Sync Migration Tools

vsts-sync-migration-tools.md

VSTS Sync Migration Tools - v7.5.x (circa 2019)

The VSTS Sync Migration tools allow you to bulk edit and migrate data between Team Projects on both Microsoft Team Foundation Server (TFS) and Visual Studio Team Services (VSTS). You can find out why this tooling exists and you can access the documentation to find out how. THis project is published as code on GitHub as well as a VSTS Sync Migration Tools on Chocolaty and VSTS Bulk Data Editor Engine on NuGet. This allows you to build your own tooling as well as just use it out of the box with a configuration file.

Quick Links from Original Author

• Video Overview
• Getting Started
• Documentation
• Contributing
• Why VSTS Bulk Data Editor
• Usage

Getting the Tools

• First option is to install chocolatey and then install the tool - this downloads to C:\tools\VSTSSyncMigration choco install vsts-sync-migrator
• Build the tool locally. Run via \vsts-sync-migration\src\VstsSyncMigrator.Console\bin\Release

Pre-Built Configuration

• See \vsts-sync-migration\configs for configuration files to use.

Prereqs

• Add the following custom field on all of the work items you want to migrate through the Administration tool: Admin > Process > Template > . Call the field name: ReflectedWorkItemId
• When you add a custom field, you can use the API to get the full name of the work item field. Example API: https://dev.azure.com/ORG/b48b8bcb-8319-4dc8-8290-ab17d5720b13/_apis/wit/workItems/4350. By running this API, we can see the full name of the work item field is Custom.ReflectedWorkItemId
• Account running is a Project Collection Administrator
• .NET Framework 4.6.1 or higher

Using the tool

• Modify a json config file to meet your needs. The most important fields to update are the source and target project.
• Run via command line (admin prompt not needed) migration.exe execute -c "path to json config
• When running via command line, pipe the output to a text file so you can reference later.
• Example command: migration.exe execute -c "C:\Users\jjs1722\Source\Repos\vsts-sync-migration\configs\vstsbulkeditor - only WITs and Tags.json" > output.txt

Starting Over

• For when you inevitably mess up and need to start over, follow these tips:
  • Write a query in the target project and CTRL+A --> Right Click --> Delete work items. This will send them to the trash. You can then delete from the trash. More info on trash can
    • For Test cases/plans/suites, you will have to use C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\witadmin.exe destroywi /collection:https://dev.azure.com/ORG /id:XXXXX,XXXX to destroy.
  • In source project, write a query that displays the work item critieria that you were trying to replicate. Select all of the work items, right click, and Edit. Bulk edit the ReflectedWorkItemId field to be nothing - ie: empty text box.
    • Note: Sometimes this didn't work for me, so what I did was changed ReflectedWorkItemId to something like "1" and then change to: "" (empty)
  • This is much less destructive than using VstsSyncMigrator.Engine.Configuration.Processing.WorkItemDeleteConfig. Of course, if you are migrating things like nodes, queries, etc., it might be easier just to delete the target team project and start over, if possible.

Configuration Notes

• some documentation can be found online on the processors from 2018 here
• Source and Target: Update source and target projects and URL's. Example: https://dev.azure.com/ORG . Also, add the source and target project names (spaces are okay to be used).
• WorkItemTypeDefinition: Add any work items you expect to migrate. If you were migrationg from Scrum to Agile, you would map Product Backlog Item to User Story
• FieldMaps:
  • Not in most of the pre-built configs. See \configs\configuration - fresh from sept 2019.json for fresh copy.
  • By default, when you are moving from source to target the system will map all of the fields that exist in source to the same field in the target. This is the behaviour if the FieldMaps section is not present in the configuration file.
  • However sometimes you want to move data to another field, or use a regex to parse out just the bits that you want. To help we have built a number of mapping tools that should give you the versatility you need.
  • VstsSyncMigrator.Engine.Configuration.FieldMap.FieldBlankMapConfig : don't use the asterisk to map to Custom.ReflectedWorkItemId because it will write to that field and then overwrite right away. We need the reflected work item there to do the attachment adding if I recall.
  • VstsSyncMigrator.Engine.Configuration.FieldMap.FieldValueMapConfig : you can map each work item type from the old state fields to new state fields. See AEA-PlatformDevops-configuration-ALL.json for an example on this. You can add more than one, which is useful because different work items have different states.
  • VstsSyncMigrator.Engine.Configuration.FieldMap.FieldMergeMapConfig : you probably don't need the default settings / get rid of it otherwise the acceptance criteria will be duplicated into description.
  • VstsSyncMigrator.Engine.Configuration.FieldMap.RegexFieldMapConfig : Not sure
  • VstsSyncMigrator.Engine.Configuration.FieldMap.FieldValuetoTagMapConfig : Not sure
  • VstsSyncMigrator.Engine.Configuration.FieldMap.FieldtoTagMapConfig : this will tag the original State to the new work item. This is useful when switching from Agile to Scrum. See: AEA-PlatformDevops-configuration-ALL.json
• Processors:
  • VstsSyncMigrator.Engine.Configuration.Processing.WorkItemMigrationConfig: The most common processor to use. This moves all work items from source to destination. Keeps original creator (note on original createdBy: If you query, it shows right createdBy by mimicking username, but if you open work item, the person who ran the tool's picture will show), rich text formatting, state, tags, area, etc. Test comments get migrated as well; they get rolled up into a singular comment along with all of the original work item information (ie: original ID). Everything but original date created and work item history.
    • PrefixProjectToNodes: This adds a prefix based on the project name. Example, if source Team Project was AAS and subarea was TFS, the target team project's structure would be AAS_2\AAS\TFS. This value has to be the same in all of the processors for things to work smoothly.
    • Stats: 2:30 minutes for 33 work items
  • VstsSyncMigrator.Engine.Configuration.Processing.WorkItemRevisionReplayMigrationConfig: I don't enable this - But if you NEED all history, this re-plays the work items with all of the changes, in order. This processor is mutually exclusive to VstsSyncMigrator.Engine.Configuration.Processing.WorkItemMigrationConfig
    • PrefixProjectToNodes: This adds a prefix based on the project name. Example, if source Team Project was AAS and subarea was TFS, the target team project's structure would be AAS_2\AAS\TFS. As previously noted, this value has to be the same in all of the processors for things to work smoothly.
    • push all revisions of work items from one location to another while maintaining context. Make sure that you add a ReflectedWorkItemID and you can restart the service at any time. This context will move replay all the revisions of work items in the source system in the exact same order on the target system, thus resulting in the very same history and state graph on the target system.
    • Stats: 15 minutes for 33 work items
  • VstsSyncMigrator.Engine.Configuration.Processing.WorkItemUpdateConfig: In theory, this should use the ReflectedWorkItemId to keep work items in sync. ~~~This also seems to need to be enabled if you want to keep original author. ~~~
  • VstsSyncMigrator.Engine.Configuration.Processing.NodeStructuresMigrationConfig: Creates areas and iterations in target project based off source.
    • PrefixProjectToNodes: This adds a prefix based on the project name. Example, if source Team Project was AAS and subarea was TFS, the target team project's structure would be AAS_2\AAS\TFS. As previously noted, this value has to be the same in all of the processors for things to work smoothly.
  • VstsSyncMigrator.Engine.Configuration.Processing.LinkMigrationConfig: Migrate external and related links (includes child/parent).
  • VstsSyncMigrator.Engine.Configuration.Processing.WorkItemPostProcessingConfig: No idea what this does. I don't enable.
  • VstsSyncMigrator.Engine.Configuration.Processing.WorkItemDeleteConfig: Woops... Can I just start again? Feed this a query and watch those items vanish
  • VstsSyncMigrator.Engine.Configuration.Processing.AttachementExportMigrationConfig: Exports all work items attachments to the migration machine. This is used in partnership with the AttachmentImportMigrationContext
  • VstsSyncMigrator.Engine.Configuration.Processing.AttachementImportMigrationConfig: Imports all work items attachments from the migration machine. This is used in partnership with the AttachementExportMigrationContext
  • VstsSyncMigrator.Engine.Configuration.Processing.WorkItemQueryMigrationConfig: Exports shared queries
  • VstsSyncMigrator.Engine.Configuration.Processing.TestVariablesMigrationConfig: Migrates test variables
  • VstsSyncMigrator.Engine.Configuration.Processing.TestConfigurationsMigrationConfig: Migrates test configurations
  • VstsSyncMigrator.Engine.Configuration.Processing.TestPlansAndSuitsMigrationConfig: Migrate test plans and suites, this does assume that the actual test cases and shared steps are migrated using the WorkItemMigrationContext
  • VstsSyncMigrator.Engine.Configuration.Processing.TestRunsMigrationConfig: BETA: Migrates past test run results
  • VstsSyncMigrator.Engine.Configuration.Processing.ImportProfilePictureConfig: Assuming not important.
  • VstsSyncMigrator.Engine.Configuration.Processing.ExportProfilePictureFromADConfig: Assuming not important.
  • VstsSyncMigrator.Engine.Configuration.Processing.FixGitCommitLinksConfig: Allows you to fix the migrated Git commit hooks (and thus external links) to point to the new repository in the target project. If the source and target repository names are the same, this will work out of the box. If the target repository has a different name, you can specify that name via the "TargetRepository" property.

Tips

• Edit the json to have the processors in the order you want them to run. Ie: Make sure the nodes (areas and iterations) processor runs before the work item import.
• Keep the "PrefixProjectToNodes" property the same for all processors. I prefer to have it = true and you can clean it up after the migration run. This will create the nodes like: TargetProject\SourceProjectName\SourceProjectSubArea. When you go to delete the areas in the target project after the migration, it asks where you want to move all of the items under it to.
• In the "WorkItemMigrationConfig" processor, write the QueryBit to support your migration. Ie: You can specify a certain subarea or tag.
• Recommended processors for full migration (incl test cases): NodeStructuresMigrationConfig, WorkItemMigrationConfig, TestPlansAndSuitsMigrationConfig, LinkMigrationConfig, AttachementExportMigrationConfig, AttachementImportMigrationConfig, WorkItemQueryMigrationConfig.
• You can do a try run by pointing the target project to a test project. It is especially important to ensure the WorkItemMigrationConfig's "UpdateSoureReflectedId" property is set to FALSE. I usually have this set to FALSE anyway to lessen the impact of email notifications sent to users due to the work item being updated. This also is the best for leaving the source project in the original state with last modify dates intact.