BLOG: Implementing Continuous Cloud Validation

Title: Implementing Continuous Cloud Validation

openstack-ci-testing-drawing

Title: Implementing Continuous Cloud Validation

How we built a cloud based CI Platform for our OpenStack Network Testing Solution. This is the story of a real-world technical implementation of a highly parallelized OpenStack CI (Continuous Integration) process.

Spirent is the leader in the test and measurement industry. As more of our customers are leveraging Openstack our products now support network testing on cloud based infrastructure with multiple hypervisor flavors.

One of the challenges in testing an open source cloud based infrastructure is the rapidly changing codebase. There are multiple patches being released daily with different application, operating system, and cloud management distributions.  We will discuss how our testing process has adapted taking all of this into consideration.

Like many organizations our developers had provisioned various adhoc Openstack installations based on DevStack with differing levels of support and documentation. What worked for one application on one development environment didn’t function when deployed to the test environment. How were we going to deal with customers wanting to deploy or solutions on Havana or Icehouse or working on products for the upcoming Juno release?

We performed an assessment and gathered requirements to provide our developers with on-demand, multi-node Openstack test environments where they can, in a whiteboard like fashion, configure different network functions, change them across all different types of distributions – and all of this would have to integrate with our existing build process for our products. The main objectives were ease and speed of test lab setup, security, repeatability, and cost.

Our test rigs perform both functional and performance tests as well as probing for security weaknesses and vulnerabilities.  As such we need to push changes to a variety of environments with multiple physical and virtual components.

For performance testing we use a cloud based test lab hosted at Nephoscale that has access to the latest Spirent traffic generation equipment:

• N4U Chassis with both DX2 and MX2 lines cards configured with 40gbps using QSFP+ break out cables running the Spirent Test Center Layer 2-3 solution.
• C100MP Chassis with 10gbps SFP+ ports used for Avalanche and Avalanche Next tests.
• Landslide for mobility use cases.
• This test harness uses the MRV Layer 1 switching solution to build the needed topology connecting test ports to bare metal servers running VMware ESXi, RedHat KVM, and Microsoft Hyper-V hypervisors.
• These hypervisors are managed by OpenStack solutions deployed with a combination of Mist.IO, SaltStack, Heat templates, and RDO Packstack.
• A standard set of virtual machine images are deployed here with http URLs from our Box share maintained by a community team of developers.
• Nephoscale is billed on a monthly basis so infrastructure changes here are less frequent and we plan for usage on a quarter by quarter basis.

For functional and regression testing the nested hypervisor capabilities provided by Ravello Systems are leveraged to run KVM and ESXi Virtual Machines on AWS.  With their “blueprint” technology and API SDK we have empowered our developers to build their own multi-node Openstack lab environments in the cloud with different distributions at the push of a button. Ravello offers all the functions of our physical test lab yet only uses virtual machines and networks deployed on a public cloud infrastructure. Machines typically only live for a few hours at a time and we can scale out to hundreds of machines depending on the combination of operating systems and networks needing to be tested.

Mist.io is used in order to provision the OpenStack host machines through the NephoScale API, to kickstart the RDO deployment process and to provide realtime monitoring of the testing procedures. Custom metrics are configured both on the host machines, as well as on the guest VM’s. These trigger alerts and automated responses when needed, like the provisioning of additional compute and network nodes. Mist.io also provides a unified dashboard for managing our multiple OpenStack installations (Havana, IceHouse, Juno).

Since some of the products we’re testing include our sensitive Intellectual Property (IP such as our virtual test appliance images), we must implement strong security controls. Our Openstack test labs running on Nephoscale and AWS through Ravello are completely fenced off from both external access and each other. This allows us to implement parallel test labs for our customers as well with minimal concern for data leakage.

Spirent’s Velocity suite of iTest products is used for automation and orchestration of Test Element Provisioning, Test Execution, Report Generation, and Data Analysis.

At a high level, our new development and Continuous Integration (CI) Testing process looks like this:

1. Developers make changes to our product codebase and commit those to on our own locally hosted GIT repositories for source code version control.
2. 1. When a task is being worked on the developer performs a git pull to obtain all the latest commits made by other team members. A task should not exceed 6 hours. This ensures that a developer publishes their changes as soon as possible ensuring that others have access to the most recent code changes.
   2. Maven is used as mvn clean install which makes a build locally on the developer’s workstation to verify all is well. This is where the developer’s unit test will be performed.
   3. The files are added to the git repository, the changes are committed with some comments about the task, then a final push is performed.
3. When any developer checks in some changes to the code a Git Hook will trigger a series of Jenkins jobs to perform testing on multiple environments.
4. On every commit from a developer Jenkins will trigger a build, run the unit tests, perform some basic integration and functional testing on simple test environments both in our on premise QA environment as well as in Nephoscale and Ravello. These should run pretty quick and show us right away if there’s a fundamental issue we need to address. This gives us the ability to see right away how our code will work against multiple hypervisor flavors and openstack distributions.
5. Sonar is executed as part of the build process and provides us a view on how well the source code complies with best practices, especially related to the development language. During the build, sonar validates the source code and stores the findings that are presented in a dashboard containing various reports. This shows us how our code quality improves or not in time allowing us to drill down on each violation and the details.
6. Nightly builds are also triggered by Jenkins to run unit test, integration tests, and all functional regression testing against the latest current versions of the virtual appliance images.
7. Once a week and for major releases we do all the above tests as well as regression tests on the last few previous versions of the virtual appliance images as well as some less utilized operating systems and openstack distributions.
8. The latest updated virtual appliance images are automatically uploaded to our box share to and imported into the glance repo of the OpenStack test beds.
9. These virtual appliance images are tested with various multi-node Openstack distributions such as SUSE, Ubuntu, RedHat RDO, Mirantis Fuel, Piston with different neutron networking configurations (VXLAN, GRE, VLAN, etc).
10. Our Spirent network traffic generator and analyzer appliances execute network tests on the target Openstack setup, which is provisioned, configured, and monitored by Mist.io. These tests are triggered by our Jenkins iTest Plugin which handles the test automation and orchestration.
11. Test data is collected and analyzed into a data warehouse. Not just the performance and security findings from our traffic generators we correlate these metrics with infrastructure utilization such as CPU, RAM, PCI Bus, Network, Storage, and Power.
12. OpenStack test environments are automatically shut down and the cycle continues.

With Ravello we first save the existing Openstack setup as a Blueprint, and then apply any new patches or configuration changes that are needed. This allows us to go back and run instances of Openstack from existing blueprints and compare those with environments where the new patches were applied.

These blueprints are treated as code with versions and we use a URL to the blueprint when we file defects in our JIRA issue tracking system from Confluence.

In summary, we have instrumented a workflow to continuously test our products with Openstack without the hassle of having enough hardware and spending time provisioning multi-node Openstack environments with different flavors for each developer. Each of our developers can spin up their own multi-node Openstack labs very quickly, run them on AWS, execute their automated tests, even run parallel tests on multiple Openstack setups and shut them down, when they are done. This new workflow process has improved quality and allows us to support multiple distributions used by our customers.

---

About the author:

https://www.twitter.com/iben

Iben Rodriguez

As Spirent’s Principal Cloud and Virtualization Architect Iben Rodriguez is working to shift network testing functions out of the test lab and closer to the developers and operators of Enterprise Data Centers. Trained in Agile, ITIL, SOX, PCI-DSS, ISO27000 he develops Testing Solutions for Businesses that leverage Cloud Technology.

Iben started his security career maintaining the data communication systems used by military reconnaissance, followed by stints in the global pharmaceutical and fabless semiconductor industries. Over the past 10 years, Rodriguez has assisted executive teams from both enterprise and government organizations in his roles as a cloud solution strategist and security architect at leading companies such as CA, VMware and eBay.

As a volunteer for the Center for Internet Security, Rodriguez initiated and maintains the Benchmark Standards for hardening hypervisors used for virtual workloads. As part of the Virtualization Security Roundtable podcast series, he also works closely with vendors to stay ahead of the latest products and trends in cloud computing.