onsdag 25. september 2013

DPM 2012: Backup and Recovery of a Physical Windows Server 2008 R2

Before starting

Verify that you have the Windows Server Backup feature installed on the target system and that there is enough space on the disks. For more details, look here.

When restoring, the drive you restore to must be the same size or larger. Move all software and data that is not part of the OS to another drive, if possible. Consider to shrink the system partition in Computer Management > Disk Management (leave 15 GB free space). To shrink it you may have to use a third party tool like PerfectDisk, and perform a free space consolidation defragmentation, before trying to shrink the volume.

Remember that a Bare Metal Recovery backup do not include any data disk, you need to backup those separately.

If your system is using UEFI to boot instead of BIOS, then a copy of the Boot Configuration Data could come in handy if you have to recreate the boot configuration. To take a copy of BCD use the command:
bcdedit /export masterBCD

Backup without DPM

You can create a Bare Metal Recovery backup directly on the target computer using wbadmin from the command line like this:
wbadmin start backup -backupTarget:\\server\bmrbackup$ -allCritical -systemState -vssFull

In this example, the backup target is a UNC path. The backup target cannot be included in the backup, but it can be on a local drive, including removable storage like a USB drive. After creating the folder, you need to share it (even if it is local) and set permissions so that the user running the backup can write to the target folder. Remember to modify the share permissions.

If you receive errors, always look at the event log first for any related errors. You can also look at the backup logs in the folder %windir%\Logs\WindowsServerBackup. The logs are in a binary format so you have to convert them before you can read them. This command will do the job:
tracerpt C:\Windows\Logs\WindowsServerBackup\Wbadmin.0.etl -f HTML -report Wbadmin.0.htm

If the backup catalog is corrupt, you could try to delete it with the command:
wbadmin delete catalog

Backup with DPM

In DPM Console > Protection > New

DPM is not able to calculate Disk Allocation for a System State backup. We need to adjust the size. For details look here. In short, we calculate the Replica Volume size with the formula:
(Data source size x 3) / 2

And we calculate the Recovery point volume size with formula:
(Data source size x retention range in days x 2) / 100 + 1600 MB

In this case we have a System Volume (C:) with 147 GB of data, so we use:

This can take a long time, and you will not see any Data Transfer in DPM because the backup is running locally on the target server. However, WSB will create the shadow copy directly on the DPM replica volume over the network. If you have a slow network, it would be considerably faster to backup locally using the wbadmin tool.

If you want to see the progress of the system state backup, go to the target server and start Windows Server Backup.

If you see a new Backup message like this, you can double click it to see the progress:

In there you will also see that the backup location is on the DPM server.

Troubleshoot the backup

You can look in DPM Event Logs on the DPM server for any related errors.

Two common errors in the DPM Alerts log would be Event ID 3106 and 3100, both indicating that DPM is running out of disk space. You would need to make the replica volume size larger.

Another place to look is in the WSB Backup logs as described earlier (see Backup Without DPM).

You can also look in the event log on both DPM server and the target server for related VSS errors. On the target, you can run commands to list the VSS writers and shadows, like this:

To see if any writers show up in a failed or hung state you can run vssadmin list writers (make sure the ASR Writer is in a healthy state, the Waiting for completion state is normal):

To see if any snapshot is in progress use vssadmin list shadows:

Another root cause to investigate is the Page file allocation on both target and DPM server. For more details look here.

You can also look at the DPM logs in the DPM installation folder (by default %Program Files%\Microsoft DPM\DPM\Temp). Look at the MSDPMCurr.errlog for conflicting jobs by other applications etc.

Restore a Bare Metal Recovery backup using DPM Console

When we want to do a restore, we start by restoring the Bare Metal Recovery from backup to a network share (or you could use a USB drive), preferably on the DPM server itself (faster restore). The main point if you want to restore over the network, is that we need a folder that is shared and accessible to the target server. We can also restore to a USB drive that we can attach to the target server later.

Verify that a Bare Metal Recovery exist in DPM Console > Protection:

Next, we restore using DPM Console > Recovery, to a folder with enough space, select the Bare Metal Recovery and then click Recover:

After the Disk recovery job has completed, we need to share the folder (or copy the WindowsImageBackup folder to a USB drive). Locate the folder that contain the WindowsImageBackup folder:

Then open Right click and open Properties > Sharing > Advanced Sharing

The default Permissions, everyone can Read, is sufficient, verify this in Permissions.

Restore a Bare Metal Recovery Backup on the target computer

To restore to the target server you need the Windows Server 2008 R2 installation media. Boot the server from this and select the Option Repair your computer:

Depending on your hardware, you may have to load drivers for your raid controller or storage controller. Download the latest drivers from the vendor to a USB drive, insert it and load the drivers, and then remove the USB drive to avoid confusion as to where the restore should go.

Restore using GUI

To recover from your bare-metal backup, select Restore System Using a System Image You Created Earlier, and click next. If you did not insert a storage device containing the system image backup, you will see the message: A valid backup location could not be found, select Cancel:

In the next dialog, select Restore a different backup:

Click the Advanced button:

Then Search for a backup on the network and click Yes to confirm:

This will require that you have a DHCP server on your network. If you do not then you can press Shift+F10 to open a command window and type:
set address "Local Area Connection" static 1

Modify set address to match your adapter name and the network address you got from the ipconfig command. Make sure you do not use an address that is already in use on your network.

After you have connected to the network, you must specify the location of the backup (UNC path to the share you created earlier): \\DPMSERVER\BMRRestore
When you click OK you must supply credentials. Make sure you type in the domain or machine name in addition to username (\\CONTOSO\user).

If authenticated, you will see the backup location, select it and click next. Then select the backup and click next.

If you select to Format and repartition disks, remember to exclude any data drives to avoid data loss:

After this you will see a summary screen, click Finish. Then you will have to Confirm the restore. If any error comes up, you can use the command prompt to investigate further.

One error that you may see is the message: No disk that can be used for recovering the system disk can be found. If you see this, then verify that you see the disk by starting command prompt and type:
list disk

If you do not see your disk then you may have to load drivers. If you see more than one disk then consider to remove all disks that is not your system disk. If you see your disk, then list the volumes by typing:
list volume

If you see any volumes on the system disk, you can remove them (this will erase all data on the disk so be very sure you have the correct disk before you do this) by tying:
list disk
select disk 0

Replace the disk number 0 above with the correct number for your system disk that you identify using the command list disk.

After this try to restore again.

If you still get the error, then perhaps your old disk was bigger than the new disk. Obviously getting a bigger disk will resolve this, or, if the new disk is bigger than the system partition on the original disk, you can use the command line to do the restore.

Restore using command line

This also requires you to boot from the Windows Server 2008 R2 installation media and select the Option Repair your computer. Load driver if needed and then select the option Use recovery tools that can help fix problems starting Windows. Then you need to copy the WindowsServerBackup folder from the DPM restore to a USB disk. Verify that you have the folder WindowsServerBackup in the root of the disk (e.g. F:\WindowsServerBackup). If it is in a subfolder (e.g. F:\BMRBackup\WindowsServerBackup), you can move it using the command line it by typing:
move F:\BMRBackup\WindowsServerBackup F:\

To do a restore using wbadmin you need to find the Version identifier for the backup. To get it type:
wbadmin get versions -backupTarget:F:

Where F: is the USB disk where you have the WindowsServerBackup folder.

You will now be able to find the Version Identifier and can copy it (left click highlight, right click highlighted text). Next step is to verify the backup by getting a list of items in the backup:
wbadmin get items -backupTarget:e: -machine:myserver -version:<right click to paste copied text>

Before you do a restore you should always now about the disks in your system. To see them type:
wbadmin get disks

If you see any data disks that you need to protect, you can copy the Disk identifiers from the list and use them to exclude the disks from the restore.
For example, if the backup is on drive F: and the Version identifier is 09/25/2013-11-18 and we have two data disks with Disk identifier {bb33dddc-7ee2-4433-abab-22fe3f471ccc} and {ee33dddc-7cc2-4433-abab-22fe3f471cee}, then the command to start a restore would be:
wbadmin start sysrecovery -version:09/25/2013-11:18 -backupTarget:F: -recreateDisks -excludeDisks: {bb33dddc-7ee2-4433-abab-22fe3f471ccc},{ee33dddc-7cc2-4433-abab-22fe3f471cee}

If the restore was successfull you can now restart your system and hopefully it will boot up and work as expected.

Restore using command line - take 2

If, for some reason, you are unable to use the sysrecovery command, you can try the start recovery command. However, this may require you to manually create the partitions before the restore. In addition, you may have to restore the boot configuration.

If your system use UEFI it will help if you have a copy of the Backup Configuration Data. This can usually be taken from a similar configured server. To get this file, go to another system that use UEFI and open command prompt, then type bcdedit /export masterBCD. Copy the file masterBCD to the USB drive.

To create the required partitions on the target system, open command line and type:
list disk         (note down the disk number for the system disk that you will restore to)
select disk 0     (replace 0 with the correct number)
clean             (PS! all data will be lost)
convert GPT
create partition EFI size=100
format quick FS=FAT32 label=System
create partition MSR size=128
create partition PRIMARY
format quick FS=NTFS label=Windows
assign letter=C

If any other volume use letter C, say volume 0, then you can change it to another letter like this, before you assign letter=C above:
list volume
select volume 0
assign letter=G

Now we can use wbadmin to restore the backup to volume C. For example, if the backup is on drive F: and the Version identifier is 09/25/2013-11-18, then the command to start a restore would be:
wbadmin start recovery -backupTarget:F: -recoveryTarget:C: -itemtype:Volume -items:C: -version: 09/25/2013-11-18

After the restore, we probably need to fix the boot configuration:

If your system use UEFI to boot

Import the Boot Configuration Data we exported earlier, from command prompt type:
bcdedit /import F:\masterBCD
bcdboot c:\Windows

Assuming that you have the masterBCD file on drive F:

From the output of bcdedit, look at Windows Boot Loaders and copy the resumeobject value for the one with device unknown, it should look something like this: {b4222de1-42e1-1112-9889-0134e3b4c221}. Then type:
bcdedit /delete <copied value from above> /cleanup

Then remove all removable storage, cd/dvd’s etc and restart.

When you start the system, you can choose to start normally if you get the message: Windows did not shut down properly.

If your system use BIOS to boot

Open command prompt and type:
bootrec /fixmbr
bootrec /fixboot
bootrec /rebuildbcd

If the above is not enough, you can try:
bootsect /nt60 c: /force /mbr
bcdboot c:\windows /s c:

mandag 23. september 2013

OpsMgr 2012: Exchange Server 2010 Managment Pack missing Reports

Just a small tip if you have the Exchange Server 2010 Management Pack installed and for some reason the Reports are missing. I have seen this happen when a reinstallation of reporting services was performed.

To resolve it, try this:
  1. Verify that you have a copy of the, if you do not you can download it from Microsoft here:
  2. If you download the management pack, verify that it is the same version that you already have imported. Also, you should not install the management pack, use a tool like 7-Zip to extract the file
  3. In Operations Manager Console, delete the existing Exchange Server 2010 Reports Management Pack.
  4. Open the Reports page used by Operations Manager in a browser, it it typically http://localhost/Reports if you are on the reports server.
  5. If you see a folder called Microsoft.Exchange.2010.Reports, delete it.
  6. In Operations Manager Console, import

torsdag 19. september 2013

OpsMgr Alert: Partitioning and grooming has not completed

When you see this alert it could be that you only have one management server and it has been offline for more than a day. In that case the alert will close next time the monitor runs, by default at 12:01 every day.

But if you have some problem preventing partitioning and grooming, you should get more details fast, so:
  1. Open SQL Server Management Studio with sufficient permissions to run a Stored Procedure against the OpsMgr database, connent to the instance that holds the OpsMgr database.
  2. Right-click the OpsMgr database > New Query. Type:
  3. Hit F5 to run this Stored Procedure.
  4. Within a few minutes you will se the result under Messages. If you see any errors you will probably be closer to understand what to fix...
More on this monitor on MPWiki:

For a more detailed explanation on the Partitioning an grooming process, take a look at Kevin Holmans blog on the topic: OpsMgr 2012 – Grooming deep dive in the OperationsManager database

In short he recommends:
If you ever have a problem with grooming - or need to get your OpsDB database size under control - simply reduce the data retention days, in the console, under Administration, Settings, Database Grooming. To start with - I recommend setting all these to just 2 days, from the default of 7. This keeps your OpsDB under control until you have time to tune all the noise from the MP's you import. So just reduce this number, then open up query analyzer, and execute EXEC p_PartitioningAndGrooming When it is done, check the job status by executing select * from InternalJobHistory order by InternalJobHistoryId DESC The last groom job should be present, and successful. The OpsDB size should be smaller, with more free space. And to validate, you can always run my large table query, found at: Useful Operations Manager 2007 SQL queries

Also, another recommendation from Scom Skills:
Check the size of the Operational Database transaction log free space, and ensure the file is large enough to handle not only regular grooming without issue, but a significant alert storm as a buffer insurance. It doesn't hurt anything to have a transaction log sized up to 50% of your operational database :)

torsdag 5. september 2013

OpsMgr 2012 R2: OpenPegasus and cross-platform agents

In the R2 release of Operations Manager 2012 the management platform for the cross-platform agents to monitor Linux and UNIX systems will change. The current cross-platform agents use the OpenPegasus standard, an open-source implementation of DMTF CIM and WBEM.

As the Cloud-OS is making its entry Microsoft needed System Center to manage different kind of devices and abstractions, both in the cloud and physical. To overcome this Microsoft decided to use Open Management Infrastructure (OMI) to replace OpenPegasus (read more).

This also brings better performance and smaller footprint, which is important for mobile devices.

While upgrading to SCOM 2012 R2 is seamless, Management Packs that use custom OpenPegasus providers will be a problem unless they are rebuilt to be based on the new OMI framework. If you use such management packs today, contact your vendor to make sure they are ready for the new Operations Manager release, otherwise upgrading to SC 2012 R2 will render that monitoring unusable.

One such vendor is NiCE and I have already spoken to them. They have, in preparation for System Center 2012 R2, buildt OMI extension using providers:
“We are happy Microsoft decided to take OMI forward and release it in such a short time – OMI allows us to develop providers leaner and faster. We have already built OMI extensions using providers to support all our cross-platform Management Packs. As OMI is an IT standard recognized in the industry, all our customers and partners will benefit from this”.
Christian Heitkamp, NiCE Product Director
For more details, look at this press release.

Other vendors are:
OpsLogix (will also work with System Center 2012 R2)
Bridgeways (I do not know if this will work)
Quest / Dell (I do not know if this will work)
Oracle (I do not know if this will work)

onsdag 4. september 2013

SysInternals: Get rid of the Open File Security Warning

When you download files from the internet and then open the file, you may get a Warning message like this:

To get rid of that warning you can delete the stream information that is associated with that file. SysInternals has the tool to do that. Download it here.

Using Streams

Usage: streams [-s] [-d] <file or directory>
-sRecurse subdirectories.
-dDelete streams.
Streams takes wildcards e.g. 'streams *.txt'.

Example, using streams from the Pommand Prompt with Run as Administrator:
cd c:\Tools\SysInternals
streams -d autologon.exe