Alfred Da Silva

Alfred Da Silva

Study Case

The Challenge

On-Premises Backup Solution for an MSSP with Commvault™

Restyling of the Commvault environment with new backup strategies and technologies

Location
Italy

Date
September 2023

1. Restyle the Commvault environment and get rid of all the repeated failures
2. Implement the most recent Commvault features
3. Introduce a new backup strategy to meet mssp requirements
4. Reuse some of the components present in the old environment

The Commvault environment previously in use had been operational for several years. The platform was down to an outdated version and a variety of errors were being encountered in the execution of backups.

Backup Job configuration had been done prior to Commvault’s introduction of the CommandCenter Web interface, and because of this, several logics that are an integral part of newer releases were not being implemented. One of them was that application backups were not using AppAware mode. The VMs in question were undergoing two separate backup jobs: one handled the virtual machine while a second job handled the application backup, treating the device as a physical machine.
Many of the protected VMs had now been decommissioned. Therefore, there were numerous objects within the console that were no longer relevant, even generating misleading alerts in some cases.

The backup logic had been defined before the company started to provide services to their customers, officially becoming an MSSP. Thus, it did not implement an adequate segmentation between customers’ backup jobs and schedules.

The Solution

After a thorough analysis, it was preferred to start from scratch, creating a new environment, rather than fixing the previous situation. In this way, redefining backup policies, by taking advantage of the recent features introduced by Commvault, was easier and more straightforward.

We firstly evaluated the system requirements for both the Commcell machines and the storage. We decided to install a brand new physical server as the data mover (also known as MediaAgent in Commvault) and to purchase and to use a VM as the backup orchestrator (also known as Commserve in Commvault).
As for the storage, we went for a graceful transition from the previous environment. The old storage system had four 10 TBs LUNs to store the data and all of them were almost full. We created a new 10 TB LUN in the already used SAN to use during the transition. The idea was to put a subset of data there and delete the same from the original environment, thus freeing up space in the older storage, then reuse the freed space for the new Commvault environment. In this way, we were able to have the two environments coexist during the transition period by just adding a new LUN, without purchasing new storage.
We also connected the new installed MediaAgent to the SAN where the virtual infrastructure resides, so as to leverage the SAN transport mode and make the data transfer as fast as possible.

Then, we added the first data to the new environment. We started with the Microsoft 365 suite. In the last years, Commvault introduced new integration functionalities in the Command Center web GUI which streamlined the process of adding 365 apps to the protected environment. Commvault utilizes Azure enterprise Apps and APIs to manage the backup process of the 365 environment. It offers a guided setup process (which is the suggested option in a standard configuration), called Express Configuration, which basically only requires providing a 365 administrator account with MFA disabled and carries out all the setup process. In the backend, Commvault creates an Enterprise Application, Requests and Grants Permissions to Azure APIs and Create a client secret for the application. Then it enters the Application ID, Directory ID and Client Secret values into Commvault to complete the setup. We preferred doing this tasks manually since we did not want to disable the MFA for an administrative account.

Next, we moved to VMs. We defined various protection groups (also known as VM groups in Commvault) to organize the machines. Each customer could have up to two protection groups: one for the standard backup and another one for the application consistent backup, if required. In addition, we also create protection groups for the provider’s internal machines, with the same logic. By doing so, it would be easier for the MSSP technician to manage the VMs based on the customer’s subscriptions. The groups were base on the tags that we had previously defined and applied inside VMWare vCenter. Each VM had two tags applied: the first indicates the customer it is related to, while the second one indicates the backup type. This allowed us to have dynamic groups that can protect also the VMs that will be deployed after the end of the Commvault project.

Application consistent backup in Commvault requires the installation of a software package inside the guest OS. To ensure the execution of backups, we also had to set up network policies into the MSSP infrastructure to allow the required traffic.

At that point, we were backing up all the assets that were present in the previous environment. However, we wanted to further enhance the backup resiliency. We decide to add two additional copies that were not configured previously, the first one would be retained in the Commvault Metallic Cloud Storage Service (MCSS) while the second would be retained in a Tape Library storage. The data stored in the cloud are the critical ones, with a low retention in order to maintain the cloud costs as low as possible. In the other hand, the tape copy serves as a long-term retention copy and will leverage Commvault vaulting feature to streamline the process of tape management and to ensure a complete protection against ransomware attacks.

After setting up the backup process, we made the backup console accessible to MSSP backup administrators. We connected Commvault to the local Active Directory domain and imported the necessary users. After that, we defined custom roles as requested by the customer and applied them to the previously imported users. Then we also configured and tuned notifications to enable operators to detect failures in a timely manner and intervene if necessary, avoiding accumulation of multiple errors.

Results

💡 Upgrade Commvault to its latest version and implement the newer features
Commvault is now running at the latest release available at the time of writing. The use of Command center was fundamental to configure backups for the Microsoft 365 suite and also the AppAware functionality is now part of the backup strategy.
💡 Design a backup solution that makes it easy to manage customer’s subscriptions
The new backup strategy organizes the protected assets into groups based on the corresponding customer and on the protection level agreed in the SLAs. Deleting data related to a specific customer or adding a new VM to the protected ones is now a very simple activity
💡 Minimize hardware expense
We were able to reuse the storage system that was used in the previous environment. The project required just a new physical server in substitution of the older MediaAgent, which was running on outdated hardware and needed to be replaced
💡 Improve redundancy and ensure ransomware protection
With the new design we were able to comply with the 3-2-1-1 rule, a best practice recognized as the standard de facto in every backup strategy: at least 3 copies of data on 2 different media with 1 copy stored offsite and 1 copy being offline, air-gapped or immutable