On my favorite FIM Portal PowerShell, Lithnet, one of the key tools is the ability to do a composite save, which is save say 500 items all at once. If you have done FIM Bulk updates you will find out that the big critical path of the whole operation is the serial save, it can take hours to update just one attribute. I once did an update to a single attribute on 30,000 objects and it took 12 hrs. With Lithnet composite save you can do a single save but 30k objects is too large, you see on the back end Lithnet creates an SQL script with all the objects, the larger the number of objects, the larger the SQL query which is definitely time out as you run out of resources. So one has to do small batch saves.
There is also another task I wanted to do. The composite save is easy to use when you do an XPath to pull the objects. So you can
$Objs = Search-Resources -XPath “/Person[OfficeLocation = ‘New York’]”
foreach($obj in $objs)
{
$obj.Location = “New Jersey”
}
Save-Resource $objs
But what about when I want to read data from a file and make a composite save. So we want
- Read items from a file, combine them and make composite save
- Save in batches of 200 (I have found this number to be the best workable batch number). The number depends on a lot of factors, the memory on the SQL server, how busy is the SQL server, how many attributes are you changing. I have done composite save in batches of 600 and even 1000 before. But on one script i was updating 8 attributes and I could not even do a batch of 200, I had to drop to 100. Again factor in memory on the SQL server, how busy is it, how busy is your FIM, do your changes trigger off other jobs which increase load on the SQL server? If you had say 80GB RAM on a dedicated FIM Service SQL server then you can probably do batches of a 1000 easily.
Solution
With this solution below the same operation that took 12hrs to update took 45minutes.
$count = 0;
$objsToSave = @();
$csv = Import-CSV C:\myfile.csv
foreach ($item in $csv)
{
$count++;
$obj = Get-Resource -ObjectType Person -AttributeName AccountName -AttributeValue $item.AccountName
$obj.OfficeLocation=$item.OfficeLocation
#Combine the items
$objsToSave += $obj;
if ($count -gt 200){
# We have more than 200 objects, save them as a group
Save-Resource $objsToSave
# Empty the array
$objsToSave = @();
# Reset count to 0;
$count = 0;
}
}
# If we have any items left after the csv has completed, save them now before we exit
If ($objsToSave.Length -gt 0)
{
Save-Resource $objsToSave;
}
Note: Composite requests cannot be used to modify filter or the behavior of Sets or groups, user single target requests. You will get this error message if you try and use composite requests to modify Filter or a system attribute (like Deferred Evaluation) of a set or group.
tlktechidentitythoughts 