Veeam Data Platform Reviews

As promised, this is the next instalment in my Veeam posts, specifically on Remote Backup (backup to a remote site across the WAN).

This has been a learning experience even though I already had one site doing remote backup for almost a year. The goal with this particular project is to leverage as much goodness as possible out of Veeam v7 for our customer. This meant learning how to make best use of some of the features of Veeam that are either new to v7 or are things I hadn’t really looked at in the past. This particular customer has Veeam Enterprise Essentials, licensing that lines up with their VMware vSphere Essentials licensing. They do NOT have the WAN Accelerator goodness that comes with the PLUS versions so it will be interesting to see how we do with traffic over the WAN.

The very first thing that I have learned from this project is to ensure the underlying VMware ESXi environment is healthy (or Hyper-V if that is your platform). An obvious statement but something that can bite you in the ass if you are not careful. And I got bit in the ass much to my chagrin. I had all sorts of weird Veeam issues happening until I sorted out the underlying ESXi issues which mostly revolved around problems with VMware Tools in the guests. All problems were solved by upping the patch level on ESXi and refreshing VMware Tools in the guests.

The next thing I learned was the Veeam documentation can sometimes be a bit “thin”. I needed to use the seeding function for backups which is NOT the same as the seeding function for replicas, at least from a process point of view. My perusal of the docs seemed to indicate that to seed a backup you need to copy a FULL backup (a Veeam .vbk file) into the remote repository (created when you create the remote Veeam agent). You then scan the remote repository in the Veeam console which should “import” the backup into the overall Veeam config, then you can use the “map backup” function to point at that backup as the seed data and you are good to go. Well, yes and no …

I tried all of that and sneaker-netted my VBK file over to the remote site, copied it into the repository then went around in 27 circles trying to get the blasted thing to scan and import. Epic fail. Nothing. I finally gave up and contacted my good buddy at Veeam, Matt Price, for some advice. It seems I was doing about 75% of what I needed to do. Matt directed me to create a net-new backup for each of the VM’s I want to have backed up to the remote site because once data is seeded at the remote site the backup job would be repointed (remapped) to the remote site. That way there would be no conflicts with the current backup job that will continue to run at the main site. He also advised me to set the remote job as a “forward incremental” with a weekly synthetic backup so that we end up with a full backup at the remote site and keep the repository somewhat pruned. So that was “ah ha!” moment number 1. “Ah ha!” moment number two was when Matt pointed out that you have to copy both the .VBK file AND the metadata file to the remote repository in order for the rescan and import to work.

So, I followed Matt’s instructions to a “T” and created my two new jobs. After they ran I copied the data via sneaker-net out to the remote repository and rescanned the repository. Voila! Things rescanned properly and the backups imported properly. I then modified each of the two jobs and used the “map backup” function to point to the remote repository backup. This effectively “seeded” the backup and I set the jobs to run. I ended up with a 50/50 status … one job ran perfectly and the other messed up.

The job that ran perfectly was for the SBS 2011 VM and it copied about 8GB of changed data out over a roughly 4 hour period (about par for the 5 megabit uplink provided by Shaw). I have to play with this backup some more in order to decide if it is going to be a daily run vs a weekly run but I am happy with the results so far.

The job that messed up seemed to mess up, once again, due to issues with the underlying VM. The Veeam job seemed to cause issues with the VM’s network stack although I cannot say why as the VM backs up properly when running the local Veeam job. I killed the running job as it had slowed to a crawl, fixed the network issues then tried re-running the job, a couple of times, as it turns out. When I finally got networking sorted out it seemed like I had totally messed up the remote “seed” backup as Veeam appeared to be trying to send ALL data back out to the remote site rather than just the changed data that I would have expected since the seed backup was taken. I have killed the whole job, deleted the job data (in local repository as well as remote repository) and am trying again form scratch. I’m hoping that I have fixed all of the underlying problems with the VM and that this job will now work as well as the job for the other VM.

So, there you have it. I’ll blog the results of my (hopefully) fixed second VM job as things progress.

I’ve managed to get a lot farther with the backup process. All of the issues with the second VM have been fixed and the remote backup of that VM has been successful. Throughput for it is the same as that for the first VM so I’m pretty happy overall.

A few things learned along the way (above and beyond what was Iisted in the last update):

• The load placed on the WAN connections can be pretty noticeable. In this case, the remote site is at an owner’s house and he definitely knows when the backup is running as his Internet connection (from machines in the house) becomes pretty “pokey”. I’m assuming the same is similar at the source end of the backup, as well. It might pay to have multiple ISP feeds to support remote backup if you are planning to move any amount of data. At the same time, it might also pay to have a separate backup network established at the source site in order to keep performance disruptions to a minimum on the production LAN.

• Veeam can gracefully recover from a failed backup run, even on first run of a “seeded” backup. My backup today was disrupted at the target end, probably when house users “messed” with things on the house LAN during backup. The “retry” feature of the backup worked well, thankfully.

• 5 Mbps as an upload speed doesn’t cut it anymore (are you listening Shaw, Bell, Telus and all the other Canadian ISP’s?). We need better upload speeds, specially for this kind of application. The upload speed will be the limiting factor in any remote backup/replication scenario. While Veeam has all sorts of tricks to help “slim” the datastream, the bottom line is there is only so much data that can be crammed through a tiny pipe over a given time period. I am truly envious of the speeds my American cousins usually enjoy on their connections. So, the extra cost of Veeam Enterprise Plus will probably be worth it for any of us that have slowish connections and/or large amounts of backup/replication traffic.

That’s it for now. I’m going to twiddle and tune the backup and will post another update once I have the schedule and such figured out.

I’ve had this remote backup in place now for about a month and a half and the results have definitely been “mixed”. In fact, I have shutdown the remote backup at this point pending moving the customer to a trial of the Enterprise Plus product that enables the WAN acceleration feature of Veeam.

I guess I should back up a bit …

The customer has standard Shaw Business service in the office and the same service at the owner’s house (remote backup site). Upload speeds on these services are 5Mbps max (burst) with, probably, closer to 4Mbps sustained being the norm. So, in practice, this really means that the most that could be transferred out over the WAN connections is about 4 ~ 5 GB per hour. This is not a large amount in terms of backup, even if Veeam is doing its dedupe and compression “thing” to make the data transfer as small as possible.

I have seen some surprisingly large transfers take place on both the SBS server VM as well as the LOB app VM and the size does not necessarily correlate with what we have seen with data updates on the servers. Apparently, other factors can also affect the actual “changed blocks” that Veeam tracks on a VM including A/V scanning activity (!!!). The net result is we have had a lot of job failures due to timeouts or other network issues related to the sheer volume of data being sent out over what amounts to a very small pipe.

This is by no means a slam against Veeam; rather, it is a cautionary reminder that there is a lot more going on “under the covers” than you might think and it highlights the reason why Veeam makes such a big deal about the new WAN acceleration feature. I’m hoping that it will allow me to overcome the inherent limitations of the “thin” Internet connection. It also serves to highlight an issue with the whole concept of “Cloud backup” (meaning backup over the Internet to wherever) in that you need bandwidth to make it all work. Regardless of the service or the technology used, in the end it all comes down to how much data you can squirt out over your WAN links in a given timeframe. All of the “tricks” used by programs/services like Veeam to thin down the data stream are all well and good but if it comes down to you having to move 20GB of data in a specific timeframe and you can only move 10, well, you have a problem.

Anyway, I’m going to bump them up to Enterprise Plus and we’ll see how it all goes. Stay tuned for the next update!

After installing Veeam Enterprise Plus and enabling the WAN Acceleration feature the size of the backups did start to drop but not enough to overcome the limitations of the single Shaw “pipes” at either end of the installation. My customer was seeing Internet performance “crawl” at each location while backups were running and, because we couldn’t get backups to “thin out” enough to run to completion within the defined backup windows, we decided to shutdown the remote backups for now. And this was also the case at the second installation where we had hoped to do replication to a remote DR site; the single Shaw pipes at each end were just not sufficient to handle the traffic generated by Veeam as well as all of the other production traffic.

I want to state emphatically that this is not a failure of Veeam; not in any way or any shape. Rather, it is a stark reminder that “inexpensive” low-bandwidth pipes can’t hope to meet the growing data transmission needs of many organizations. Bandwidth “rules” and it will make or break a project like the ones I’ve been attempting.

We are going to look into the possibility of installing additional Shaw pipes to handle only the Veeam traffic as the process itself works and there is massive value in having backup data automatically “live” in multiple locations. And we are going to pray to the network gods that Shaw will actually rollout the long-promised upgrade in bandwidth and upload speeds that we have all been waiting for for oh so long.

If you have been following my blog over the last little while you will know that I have faced a few challenges with Veeam in terms of setting up remote jobs (backups across the WAN). Well, I think I finally have some good news.

I was originally trying to do full Veeam backups of ESXi VM’s across the WAN meaning I was trying to perform a “normal” Veeam backup across the WAN. I had managed to do this with a particular customer’s VM under Veeam 6.5 and I thought the same process would work for others but I kept hitting roadblocks. At first I thought it was something to do with changes in Veeam 7.0 but it turns out to that it was far more complicated than that. As I indicated in earlier posts, the customer that I was really having problems with had fairly large VM’s with a lot of changing daily data, at least in one VM. Trying to back them up in a “normal” fashion across the WAN usually failed miserably and yet things seem to work okay with the other customer. It turns out that, 1) I was labouring under a misconception about the best way to perform Veeam backups across the WAN and, 2) it was just sheer dumb luck that backups for the first customer (the one started under 6.5) actually worked!

Now that I have gone through many more hoops at both customers as well as sorted through issues with Veeam support it seems that I may have some answers.

First and foremost, if you want to perform Veeam backups across the WAN you should be using Backup Copy Jobs as an adjunct to your “on LAN” Veeam job and NOT try to target a direct job out over the WAN (unless, of course, you have massive big pipes). The reason for this is that the Backup Copy Job is actually a backup of an existingbackup and not an actual backup of a running VM itself. What this does is backup the last completed backup of a VM and it runs between the Veeam server and the Veeam repositories and does not impact the running VM. The end result is a fully intact and verified copy (hence the Backup Copy moniker) of your backup in your remote repository. This is particularly nice as it puts no load on your production VM or the hypervisor host. This can be particularly important if you have a busy VM and it gets impacted when a snapshot is taken or removed during busy operational periods.

Secondly, if you are serious about all of this then you want to upgrade your Veeam license to Enterprise Plus so that you get the WAN accelerator feature. Backup Copy Jobs are currently the only jobs that actually use the WAN Accelerator and from what I’ve seen it can make a very large difference in the amount of data that is actually transferred across the WAN. I originally thought the WAN Accelerator worked with all types of Veeam jobs, including Replication, but that is incorrect. I have been told that Veeam is considering how the WAN Accelerator feature might be enabled for other jobs but, for now, only Backup Copy Jobs leverage the capability.

I have happily running Backup Copy Jobs, now, and the remaining hair on my head is safe from being torn out any further. My thanks to Michael Strine at Veeam Support for helping to get me pointed in the right direction and for patiently working through the whole process with me.

Disclaimer: My company is partners with several vendors including Veeam.

Before I get rolling on this post let me just say this … I love Veeam

I am unapologetic about it, I think Veeam is one of the best applications to come down the pike in a long, long time. And, no, I’m not a Veeam employee, just a very enthusiastic user of the product! OK, now for the meat and potatoes …

One thing we (itgroove) get asked about, a lot, is remote backup. I’ll be honest, to this point in time I have not come across any remote backup system that I can heartily endorse to any customer. Yes, there are a lot of remote backup services out there – some better than others – but they all tend to leave me a bit cold as they usually fall down horribly when it comes to quick recovery. Worse, they usually end up costing way more than most people bargain for. We have also tried things like replicating backup devices (we tried using the RSYNC functions built into QNAP’s, as an example) with less than satisfactory results. Frankly, I was beginning to despair of finding a system that I could trust to recommend to customers, then along came Veeam …

I’ve already waxed poetic about Veeam’s abilities in other posts so I’m going to focus on it’s abilities to both back up to a remote location across WAN links as well as handle replication across WAN links in this post. I believe that these capabilities are worth their weight in gold, specially so to SMB’s that usually don’t have the financial resources to do all the things they should in terms of backup and DR.

With the latest release of Veeam, Veeam v7, the remote backup/replication abilities have been further enhanced over the already very capable abilities of v6. The big addition is the built-in “WAN accelerator” feature available with the Enterprise Plus editions of Veeam. The WAN accelerator feature can provide some fairly impressive reductions in overall bandwidth use for both remote backup and replication as the acceleration is “tuned” specifically for these processes as opposed to general hardware appliance-based WAN accelerators that have to cope with many traffic types. The other big feature in v7 (in my opinion) is the ability to “seed” both remote replication and remote backups with full backups from your main Veeam site. This alone provides a massive jump on the WAN data bandwidth reduction side of things as you can “sneaker net” large amounts of seed data out to your remote site then let Veeam syncronize backups going forward using the seed data to start.

We have two projects underway with two different customers that will leverage these abilities and I’ll be blogging about their progress and results. I think both are of interest to SMB’s as they are both real-life cases that are applicable to many small organizations.

Project A is a project to create a reliable off site backup using Veeam to process backups of a single VMware ESXi 5.1 host and two Windows Server 2008 R2 VM’s. Veeam is currently performing on-site backups of the host with output going to a local, on-site QNAP. The remote backup process will create a second Veeam backup stream via a remote Windows 7 Veeam proxy with the backup data being streamed out to a second QNAP at the remote site. In this particular case the main site and the remote site are both using Shaw as the ISP and both sites have Sonicwall firewalls so the Veeam data is being routed over a Sonicwall site-to-site VPN link. I’ll be blogging the details in a followup post titled Veeam Backup – Project A. The overall goal is to meet the Veeam “3-2-1” backup recommendations and to have reliable backup data living in two separate locations on separate media and eliminate the need to have offsite data having to be copied to USB media and manually taken off site AND keep the costs down to a manageable level.

Project B is a project to create a “poor man’s” DR site for a customer where a critical subset of their systems is to be replicated and available on “hot standby” in case of disaster at their main operations site. I use the term “poor man’s” DR as this site is being put together on a shoe-string budget due to the nature of the customer’s finances. I’m not necessarily advocating for everyone to do something similar but it will serve to illustrate what can be done with a bit of effort and thinking even if the budget is not forthcoming. In this particular case we are repurposing some old servers the customer had along with a “found” SAN that another customer was decommissioning. While the SAN is a nice thing to have there is no reason at all why something like a multi-bay QNAP or other low-cost NAS could not be used in its place. The big enabler, here, is Veeam Enterprise Plus with the WAN accelerator. The customer’s main site has two largish VMware ESXi 5.1 hosts with multiple VM’s on each host as well as a physical SQLserver host that backends their main LOB app. The SQLserver is backed up by BackupExec and there is a weekly backup job that outputs a full VM of the server. We are hoping to incorporate that backup VM into the overall Veeam process. This one is going to be very interesting; I’ll be posting about it as Veeam Replication – Project B. As with Project A, both sites use Shaw as the ISP and both sites have Sonicwall’s and we’ll be leveraging Sonicwall site-to-site VPN connections.

So, I hope you’ll keep an eye for my updates and come along for the ride. This should be very educational, I know I’m going to learn a bunch, I hope you will, too!

Disclaimer: My company is partners with several vendors including Veeam.

Valuable Features:

Room for Improvement:

Valuable Features:

Room for Improvement:

What is our primary use case?

What is most valuable?

What needs improvement?

For how long have I used the solution?

What do I think about the stability of the solution?

What do I think about the scalability of the solution?

How was the initial setup?

What about the implementation team?

What's my experience with pricing, setup cost, and licensing?

What other advice do I have?

Which deployment model are you using for this solution?

What is most valuable?

What needs improvement?

How are customer service and support?

How would you rate customer service and support?

What other advice do I have?

What is our primary use case?

What is most valuable?

What needs improvement?

For how long have I used the solution?

What do I think about the scalability of the solution?

How was the initial setup?

Which other solutions did I evaluate?

What other advice do I have?

What is our primary use case?

What is most valuable?

What needs improvement?

For how long have I used the solution?

What do I think about the stability of the solution?

What do I think about the scalability of the solution?

How are customer service and support?

How was the initial setup?

What other advice do I have?