Archive for the ‘Virtual Server Technology’ Category
VMs that went into Saved State in the Last Day
During the migration of VMs to a Hyper-V in Windows Server 2012 R2 environment you may find that some of your VMs go in to Saved State when the backup process kicks off. This may be because you have not updated the Integration Components, there is a problem with the Integration Component install or some other issue where the Hyper-V Writers determine that the VM should be backed up offline (Look for a post on this soon). During this time of troubleshooting you can go to the Windows-Hyper-V-Worker-Admin event log on each server to identify which VMs are having this issue by looking for event 18510. If you have a larger environment this gets a little inefficient.
I have used the script below during my troubleshooting. You could easily change this to either look to a server list text file or even to accept a switch.
|$s = “”SERVERNAME1″, “SERVERNAME2″
The update of SCVMM 2012 R2 with Update Rollup one is fairly straight forward with the exception of the need to execute a SQL script after the server component update, but if you have used VMM for any length of time you know that an occasional modification to the database itself is necessary to correct issues. Even with that, the process, should take no more than five minutes to complete. Here are the steps below. Make sure you either perform this in your test environment first or wait a few weeks until others have tested the update just to be sure any unexpected issues have been vetted. There are not a lot of changes in the rollup, I can think of many more that need to be corrected, but having an outage of your main virtual machine management tool is never a good thing.
- Download the Hotfixes HERE
- Extract the MSP files from the CAB files by double clicking on the download CAB then right-clicking the MSP file and choosing extract.
3. Execute msiexec.exe /update kb2904712_vmmserver_amd64.msp
4. Execute the following SQL script: (It is stressed to do this immediately after the server side update)
/* script starts here */
ALTER Procedure [dbo].[prc_RBS_UserRoleSharedObjectRelation_Insert]
@UserOrGroup varbinary (85),
@ForeignAccount nvarchar (256),
@ExistingID uniqueidentifier = NULL OUTPUT
SET NOCOUNT ON
SELECT @ExistingID = [ID] FROM [dbo].[tbl_RBS_UserRoleSharedObjectRelation]
WHERE [ObjectID] = @ObjectID AND [RoleID] = @RoleID
— Select owner OR Select all which matches ForeignAccount or UserOrGroup OR
— both ForeignAccount and UserOrGroup is NULL
(([UserOrGroup] = @UserOrGroup OR [ForeignAccount] = @ForeignAccount) OR
([UserOrGroup] IS NULL AND @UserOrGroup IS NULL AND [ForeignAccount] IS NULL AND @ForeignAccount IS NULL))
/* Ignore duplicate entries */
IF (@ExistingID IS NULL)
SET NOCOUNT OFF
/* script ends here */
5. Execute msiexec /update kb2919248_AdminConsole_amd64.msp
|Version Before: 3.2.7510.0||Version After: 3.2.7620.0|
***You can also check out the Video put out by the SCVMM Support Team on the deployment process HERE.
The System Center Team release (KB2904734) System Center 2012 R2 Update Rollup 1 today with high hopes by admins everywhere that this will start to plug some of the large holes in the products under the System Center Umbrella released 3+ Months ago along side of Windows Server 2012 R2 as part of Microsoft’s “attempt” to not leave gaps between platform and management products. Don’t get me wrong. The System Center 2012 R2 products add a lot of great features and functionality, but between functionality bugs and features pulled that were in previous versions, it has left a bad taste in many administrator’s mouths and has gone against the goal of Microsoft of releasing capable products at GA.
Below are the links to the products that have updates. Only System Center Data Protection Manager, System Center Operations Manager and Virtual Machine Manager have applicable updates.
A 0x80070057 error occurs when a session is closed prematurely. This error is caused by a failure during a consistency check.
Lots of concurrent threads or calls to Microsoft SQL Server from the Data Protection Manager (DPM) console cause slow SQL Server performance. When this issue occurs, the DPM console runs out of connections to SQL Server and may hang or crash.
The DPM console crashes, and an error that resembles the following is logged:
An error occurs when you run the p_DataPurging stored procedure. This error occurs when the query processor runs out of internal resources and cannot produce a query plan.
Data warehouse BULK INSERT commands use an unchangeable, default 30-second time-out value that may cause query time-outs.
Many 26319 errors are generated when you use the Operator role. This issue causes performance problems.
The diagram component does not publish location information in the component state.
Renaming a group works correctly on the console. However, the old name of the group appears when you try to override a monitor or scope a view based on group.
SCOM synchronization is not supported in the localized versions of Team Foundation Server.
An SDK process deadlock causes the Exchange correlation engine to fail.
The “Microsoft System Center Advisor monitoring server” reserved group is visible in a computer or group search.
Multiple Advisor Connector are discovered for the same physical computer when the computer hosts a cluster.
A Dashboard exception occurs if the criteria that are used for a query include an invalid character or keyword.
Operations Manager – UNIX and Linux Monitoring (Management Pack Update)
On a Solaris-based computer, an error message that resembles the following is logged in the Operations Manager log. This issue occurs if a Solaris-based computer that has many monitored resources runs out of file descriptors and does not monitor the resources. Monitored resources may include file systems, physical disks, and network adapters.
Note The Operations Manager log is located at /var/opt/microsoft/scx/log/scx.log.
errno = 24 (Too many open files)
This issue occurs because the default user limit on Solaris is too low to allocate a sufficient number of file descriptors. After the rollup update is installed, the updated agent overrides the default user limit by using a user limit for the agent process of 1,024.
If Linux Container (cgroup) entries in the /etc/mtab path on a monitored Linux-based computer begin with the “cgroup” string, a warning that resembles the following is logged in the agent log.
Note When this issue occurs, some physical disks may not be discovered as expected.
Warning [scx.core.common.pal.system.disk.diskdepend:418:29352:139684846989056] Did not find key ‘cgroup’ in proc_disk_stats map, device name was ‘cgroup’.
Physical disk configurations that cannot be monitored, or failures in physical disk monitoring, cause failures in system monitoring on UNIX and Linux computers. When this issue occurs, logical disk instances are not discovered by Operations Manager for a monitored UNIX-based or Linux-based computer.
A monitored Solaris zone that is configured to use dynamic CPU allocation with dynamic resource pools may log errors in the agent logs as CPUs are removed from the zone and do not identify the CPUs currently in the system. In rare cases, the agent on a Solaris zone with dynamic CPU allocation may hang during routine monitoring.
Note This issue applies to any monitored Solaris zones that are configured to use dynamic resource pools and a “dedicated-cpu” configuration that involves a range of CPUs.
An error that resembles the following is generated on Solaris 9-based computers when the /opt/microsoft/scx/bin/tools/setup.sh script does not set the library pathcorrectly. When this issue occurs, the omicli tool cannot run.
ld.so.1: omicli: fatal: libssl.so.0.9.7: open failed: No such file or directory
If the agent does not retrieve process arguments from the getargs subroutine on an AIX-based computer, the monitored daemons may be reported incorrectly as offline. An error message that resembles the following is logged in the agent log:
Calling getargs() returned an error
The agent on AIX-based computers considers all file cache to be available memory and does not treat minperm cache as used memory. After this update rollup is installed, available memory on AIX-based computer is calculated as: free memory + (cache – minperm).
The Universal Linux agent is not installed on Linux computers that have OpenSSL versions greater than 1.0.0 if the library file libssl.so.1.0.0 does not exist. An error message that resembles the following is logged:
/opt/microsoft/scx/bin/tools/.scxsslconfig: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory
System Center 2012 R2 Virtual Machine Manager cannot deploy a new or imported VMWare template.
A virtual machine with that uses VHDX cannot be refreshed correctly in System Center 2012 R2 Virtual Machine Manager, and you receive the following error message:
Refresh job failed with error 2912: The requested operation cannot be performed on the virtual disk as it is currently used in shared mode (0xC05CFF0A)
Database operations sometimes fail with “FailedToAcquireLockException.”
A new virtual machine template from a template that specifies an operating system profile doesn’t use credentials from the operating system profile.
Virtual machines in VMWare that connect by the using Cisco N1000V dvSwitch are unavailable for management from Virtual Machine Manager.
System Center Virtual Machine Manager service crashes if you disable one of the teamed network adapters.
The Get-Scstoragearray -host command should return storage arrays that are visible to a host that is using zoning.
During the discovery of a network-attached storage (NAS) provider, the credentials that are used do not include a domain name.
Some localized strings are not displayed correctly in the UI.
A query to find the certificate should match both the subject name and the friendly name because FindBySubjectNameis a wildcard search.
Template deployment fails, and you receive the following error message:
VMM could not find the specified path on the <Server name> server.
The system cannot find the path specified (0x80070003)
Virtual Hard Disk (VHD) cannot be mounted on a host because VHD conflicts with other disks because of a stale entry that was left in the dictionary of Virtual Machine Manager memory.
Differencing disk based deployment may fail because the parent disk is being refreshed as noncached.
I would recommend getting these up and running in your test environments first. As with any update, there are chances for potential issues. If you see anything any issues, please report them back to Microsoft and help out the community.
Having worked with Hyper-V for quite some time, the debut of PHD Virtual’s Backup for Hyper-V with the v7.0 release further exemplifies the impact that Hyper-V is having in the virtualization world. If you are not familiar with PHD Virtual, they have been producing data protection and recovery solutions for VMware and XenServer since 2006, and just last week released their entry into the Hyper-V backup space. What this means for existing PHD Virtual customers that may be running VMware or XenServer is that they can work with a single vendor as they diversify their hypervisor vendors utilizing Hyper-V as many companies have started to do. What this means to those already using Hyper-V is that there is now even more options for backup and recovery. More competition is always a good thing for the consumer and the backup and recovery space for Hyper-V has been a hot playing field.
I was able to kick the tires during the early testing and have been impressed with the strait forward approach of the product and it has been refreshing to work with PHD Virtual’s development teams to address both bug and feature requests to get the product where it is today at GA. PHD Virtual’s Backup for Hyper-V has brought a unique approach to backing up Hyper-V VMs and has produced a very easy to navigate web based console to successfully protect, manage and restore your Hyper-V virtual infrastructure. Here are some highlights I saw along the way.
Installation is straight forward but does require some pre-planning. For production environments you should have a static IP Address for each VBA that you deploy and you will also need to have create a username and password for each for to security between the Management components and the service that runs on each Hyper-V Host server. This username is not a Windows username but one that is used only within the application.
Supported Hypervisors: Hyper-V in Window Server 2012 and 2012 R2/ Hyper-V Server 2012 and 2012 R2
Clicking Clicking on the installation file from the extracted download installs 4 components. Erlang, RabbitMQ, PHD Service, and the Virtual Backup Appliance (VBA). Each of these is wizard driven and there are no surprises or tough configuration choices when rolling through the screens. The last step of the install actually deploys the VBA to your Hyper-V Host including the creation of the appliance backup destination disk if necessary.
Note: The disk for the backup appliance supports all the features of Hyper-V so it can be stored on an SMB 3.0 share location as well, giving you the ability to consolidate the location of multiple appliances to a location that may allow for offsite replication.
Once you have Installed the 4 components on your Hyper-V host connecting it to the Web console is just a few clicks away. Getting the IP Address from the VBA once it is up and running and using Chrome or Firefox (IE is not supported) to connect to the address will bring you through another short intuitive wizard to connect your Web Console to your VBA. You can connect and will want to connect multiple Hyper-V hosts to your Web Console which makes most day to day functions centralized.
What Could Be Better: Installation of the Erlang and RabbitMQ Components should be silent and updated without manual intervention for future releases.
For more detailed information on the Install Process view the video HERE.
The Dashboard is presented to you as you logon to the web console and gives quite a bit of information like Errors, Number VMs protected or not protected, Storage Utilization, Deduplication ratios, Alerts and Actively Running Jobs. In my testing on smaller scales it gives some very good information at a glance that can be drilled into, but I worry how busy it might be in larger environments. It would also be nice to be able to move or add other specific consoles on the dashboard, but all the relevant overall information is there.
Note: Due to the wealth of information, looking at the console in something less than a 24 inch monitor might be a little tricky.
Before I get too deep on the process it is important to understand the method that PHD Virtual uses for their Hyper-V backup product. First off it is completely agentless to the VMs The VBA, Service and prerequisite software on the hosts, technically counts as an agent to me, but no direct PHD code is run on the VMs.
To perform the backup PDH Virtual Backup for Hyper-V uses Checkpoints (Formerly Hyper-V Snapshots). Basically, Checkpoints for each VM are created and then the data is then pulled, the checkpoints are then merged back in with no interruption to the VM. This process is possible due to the live merge feature introduced in Windows Server 2012. It will be interesting to see how this process works under heavier load as I have never had a VM Checkpoint fail, but have had numerous fights with the Hyper-V VSS Writer over the years. In the current 7.0 release of PHD Backup for Hyper-V, VSS is not utilized to quiesce the VMs during backup. As a result you have Crash-Consistent backups meaning all your files will be backed up at the same time, but transaction logs would be lost. Application level backups are on their way as you can see by the screen print below.
Agentless Backup: Not many successful VM backup products use agents on VMs, but instead use Host Based backup process. PHD Virtual Backup for Hyper-V does this as well with a low footprint or interference with locally running processes.
Scheduling Options: All the Backup and Retention settings you expect out of a backup product are there. PHD Virtual for Hyper-V includes full backups or mixture of both Full and Incremental on a schedule set by the administrator.
Host Level Backup Selection: The option to select the top level of the host to protect all current and future VMs on that host is critical so that newly added VMs are not forgotten.
Picks most appropriate VBA: In most cased the VBA local to where the VMs are will take care of the backup process. This is configurable however at the VM or Job level.
Data Stores are configurable: VBA data drives can utilize SMB attached disk allowing you to centralize VBA disks to a location that could allow secondary backup or replication.
Dashboard About Unprotected VMs: VMs forgotten to be added to backup jobs seem to be ones that fail for some reason. There is a location on the PHD’s dashboard that tells you just how many VMs are not protected so that an administrator can remediate the issue quicker.
Deduplication: Build in Pre-Processing Deduplication of data. This will greatly reduce the space of your data store and should be a requirement of any modern backup product.
Works with replicated VMs that are utilizing Hyper-V replica: My testing was successful on backing up the primary VM of a VM replica set. No averse interaction or failures.
Works on VMs that are using SMB storage: VMs with remote SMB storage work the same as if their VM configurations files and VHD/VHDX files were local, iSCSI, or FC.
Note: Does not work on VMs that have virtual FC HBA attached disks or shared VHDX disks.
What Could Be Better:
Data Source Replication: No option for this right now leaving it up to the administrator to find creative ways to locate the Data Store to a separate location and then potentially replicate this data to another site for maximum protections.
VSS and Log Truncation: As much as I loath VSS, in order to provide true Application Consistent backups, the quiesce process needs to be sent up to the VM in order to provide ideal protection for Databases, or other memory resident applications. Without this you are relying on the built in recovery mechanism of these types of applications to recovery orphaned transaction logs or risk loosing some data. Depending on your SLA with your customers, this could be a very big deal. Good thing is, this is already on the PHD Virtual roadmap.
Hyper-V Change Block Tracking: Not really a PHD Virtual issue as Microsoft has not built this API yet, but would be nice to have compared to VMware. This would dramatically shorten backup times as changes would be advertised during the backup product instead of the backup product needing to walk all the blocks searching for changes. This is also on the roadmap for PHD Virtual, but details are unclear as this would have to be implemented separately without Microsoft at this time.
For more detailed information on the Backup Process view the video HERE.
Recovery of VMs has many of the same features an administrator would be familiar with in modern virtual server backup products aligned in a tree view console accompanied with helpful filtering options as many backups start to inevitably clutter the console view . Full Restores, and File Level Restore (FLR) are there, as well as a Rollback Recovery Option as well utilizing the Checkpoint architecture of the product and Windows Server 2012.
Full Restores: Allow you to recover your VM to the original location or to another location on the same host or alternate host with the ability to change VM attributes like Name, MAC Address, and Custom Recovery Paths.
FIle Level Restore: As the name suggests PHD Virtual Backup for Hyper-V provides a method to gather files from inside a backed up VHD/VHDX instead of having to restore a complete VM, mount up the VHD/VHDX for file level inspection. When performing the file level restore however, there is not index of files from within each VHD/VHDX. The VHD/VHDX is mounted on the VBA and you are given a UNC path to the access the files where you can then manually copy and paste them out to the original location or to what ever destination you need. It would be nice to have an index as users sometimes have no idea when they deleted a particular file, so mounting up many different VHD/VHDX files on the VBA and digging through them to see if they have the file necessary could be time consuming. The process to perform a File Level Restore could also be refined as there are a few steps to select, then PIN then Mount the FLR. In any case, the ability to do this level of restore is included and much faster than the restoration of full VHD/VHDX files.
Rollback Restore: I restoration option back to a previous point in time based on the Checkpoint feature build in to PHD Virtual Backup for Hyper-V and the architecture of Windows Server 2012. Think of it like using the standard Checkpoint technology, but not needing to retain the Checkpoint as part of the VM. On restore the VM is placed back into the previous state by merging in a previously point in time.
What Could Be Better:The interface is a little too helpful at times and adds to some confusion as you have recovery options both from the Recover sidebar option or from the Recover button within the Protect sidebar option. Both bring you to the same locations, but in my mind blurs the lines between what is usually segregated functions within most backup products. Not a big deal but it is confusing at first until you realize it is two different area that lead you to the same same recovery options.
The File Level Restore Option could be a little more straight forward as well as I stated above.
For more detailed information on the Recovery Process view the video HERE.
PHD Virtual Backup for Hyper-V comes with Reporting functions that allow you to drill down in 5 basic categories and the ability to set a desired date range . Protection Summary Report, Protected Virtual Machines, Unprotected Virtual Machines, Backup History Report, and Storage Details round out the options with the ability to Export to PDF or CSV as well as output to the screen.
What Could Be Better: A scheduling option for reports to be sent through email or published to a website would be helpful for daily reports of how last night’s jobs went to avoid having to logon to the console if no action was necessary.
With every first release product there are features I wish were there. I tend to be greedy and say, “well that’s good, but what about this?” That said, this is a solid and stable first release that is FREE. There is no reason not to give it a try. This was a great move by PHD Virtual and follows the marketing approach that Microsoft has taken itself with Hyper-V. Get the product out there with as little resistance as possible to get your product in the hands of those that really want it, the virtualization administrators, engineers and architects. Without having to request funds to test a new product, the managerial barrier to testing out all the features is dropped. It is available now for FREE. All feature, all functions, not time limits. Give it a try.
You should create a snapshot, or checkpoint (as they are called in System Center Virtual Machine Manager), right before you make any known change to your VM that would require some sort of system or application restore if the configuration, patch or application upgrade went wrong.
Snapshots are a recovery point that saves the state, data and hardware configuration of a VM. Now, With Hyper-V in Windows Server 2012, the creation and deletion of a snapshot happens without VM downtime. The process can be scripted, but it is most commonly done manually, either by an application owner or by a virtualization system administrator, before any changes are made to the VM. These types of snapshots are different from conventional host-level backups of VMs, but use the same underlying VSS Hyper-V Writer, which is installed when you enable the Hyper-V role, to prepare (quiesce) the memory and I/O state of the VM, in order to precisely capture its state at that moment. However, the outcome of these two processes differs.
When you create a snapshot… Read the rest of the article HERE
Also see.. AVHD – WT…”A”?
More demos and banter below.
TechNet Radio: ITProGuru vs. Hyper-V Live (Part 2)
- [0:50] DEMO: Live Storage Migration
- [8:57] DEMO: Live Migration
- [18:50] DEMO: Multiple Live Migrations
- [21:16] DEMO: Cluster Node Live Evacuations / Pause
I had a chance to sit down with Dan Stolts (ITProGuru) and talk about the new LIVE features of Hyper-V in Windows Server 2012 on Technet Radio/Channel 9. The content comes from my recent session at the VirtG Boston Deep Dive Day 2013, Hyper-V LIVE!. Exciting features have emerged. Demos and banter are below.
TechNet Radio: ITProGuru vs. Hyper-V Live (Part 1)
Here is the Channel 9 video location that has a nice outline of what we talked about as well as when the Demos start.