How DreamHost is reinventing itself with OpenStack

Original post came from OpenSource.com – http://opensource.com/business/14/7/dreamhost-and-openstack-love-story

Founded in 1997, DreamHost is a seasoned internet business home to over 400,000 happy customers, 1.5 million sites and applications, and hundreds of thousands of installs of WordPress, the dominant open source CMS. Open source is in our blood, and has powered every aspect of our services since 1997. DreamHost is built on a foundation of Perl, Linux, Apache, MySQL, and countless other open source projects. In our 16+ years of existence, DreamHost has seen the realities of internet applications and hosting drastically evolve. Our journey to the cloud requires a bit of history and context, so let’s dive right in.

The Rise of the Black Box Cloud

Nearly a decade ago Amazon created the market of cloud infrastructure services with the introduction of the immensely popular S3 for storage and EC2 for compute. The years that followed have been dominated by sweeping changes to the way that infrastructure is consumed and, more importantly, to the underlying design and architecture of software. There’s also there’s been a larger, hidden consequence to the rise of opaque cloud infrastructure services.

While the cloud has been revolutionary it has also been largely a black box. The software and systems that power Amazon Web Services, Microsoft Azure, and many other clouds are closed to prying eyes, leaving users in the dark about the implementation of the most critical component of their application stacks. The era prior to the cloud represented the rise of the open internet – Linux, Apache, MySQL, and languages like PHP, Perl, Python, and Ruby, where developers, engineers, and IT organizations had a large degree of transparency about the software that powered their applications. In the early cloud era much of that transparency disappeared.

A New Hope

In 2010, two unlikely partners, NASA and Rackspace Hosting, founded the OpenStack project to create open source cloud software for the creation of private and public clouds. In the years since its inception the OpenStack project has exploded, aiming to live up to its potential as the Linux of cloud. More than 200 companies and countless individuals are now a part of the project, working in concert to create open source software and APIs that power private and public clouds globally.

DreamHost joined OpenStack early in its life, committing code, financial backing, and leadership to the project. We joined the OpenStack Foundation as a Gold member, and DreamHost CEO Simon Anderson was elected to represent us on the OpenStack Foundation Board of Directors. Our commitment to the success of the project runs deep.

Why OpenStack?

DreamHost wouldn’t exist today without a strong commitment to the open source philosophy. We don’t want to live in a future that is again dominated by closed, technically opaque, “magical” cloud platforms. Many traditional hosting customers are interested in the adoption of cloud services, either in addition to, or as a replacement for, their existing shared, VPS, and dedicated hosting, and we believe that they too are looking for a simple and affordable upgrade path. Given our DNA, it makes sense for DreamHost to build our customers what they want using best-of-breed open source software.

Introducing DreamCompute

DreamHost’s first product built on OpenStack is DreamCompute, which allows customers to create virtual machines, block devices, and networks on-demand via the standard OpenStack APIs and command-line tools or via an intuitive web-based user interface. DreamCompute puts more power in the hands of our customers than they’ve ever had access to before, and is built on a large library of open source software. In true DreamHost fashion, even the architecture of DreamCompute is open.

DreamCompute runs on a mixture of high-end Dell servers running Ubuntu Linux. We have two basic types of servers: storage nodes and hypervisor nodes. The hypervisor nodes are optimized for hosting virtual machines running on top of the open source KVM hypervisor, and feature 64 AMD cores and 192 GB of RAM. Our storage nodes are lower-powered, higher-density servers, each with twelve 3 TB disks, and are running Ceph, the open source, massively distributed, fault tolerant storage system that DreamHost helped build.

DreamCompute also features a “cockpit” pod, which represents the “brain” of the cloud. In the cockpit, we run OpenStack and its supporting services on a mixture of bare metal and virtual machines, including Horizon, Glance, Nova, Neutron,Keystone, and Cinder, along with Apache, HAProxy load balancers, MySQL databases, and RabbitMQ queueing systems. The entire system is configured and managed by Chef, and is monitored using open source tools like logstash, graphite, collectd, and nagios.

Even the networking hardware and software in DreamCompute are based upon open platforms and technology. DreamHost has sourced high-performance, 48 port 10 Gig switches directly from manufacturers. The switches run Cumulus Linux, which is a Linux network operating system from our friends at Cumulus Networks. This unique setup enables us to provision, monitor, and operate our networking infrastructure using the same tools and processes that we use for our compute and storage nodes, greatly minimizing operational overhead.

DreamCompute is compatible with the standard OpenStack Compute, Network, Image and Storage APIs, and is at its core an OpenStack deployment. That said, DreamCompute also has some unique features that set it apart from other clouds. It should come as no surprise that the foundation for these features are, in fact, based upon open source software that DreamHost created.

Fear the Cephalopod

Every virtual machine in DreamCompute boots from a virtual block device backed by a multi-petabyte Ceph storage cluster. Operating system images themselves are stored in the same cluster as these block devices, enabling DreamCompute to leverage Ceph’s Copy-on-Write (COW) functionality. Rather than downloading the operating system image from a central store to a hypervisor (which is time consuming) and then provisioning a new block device, Ceph enables our virtual machines to boot nearly instantly from a thin-provisioned copy of the OS image. As a result, virtual machines in DreamCompute can be created and fully operational in as little as 40 seconds.

Ceph also provides DreamCompute users with confidence that their data is safe, as every piece of data that is stored in the cluster is replicated a total of three times. When disks, servers, or racks fail, the Ceph cluster springs into action to automatically heal itself, ensuring that the proper number of replicas exist. When new capacity is added, Ceph responds by immediately putting it to good use, rebalancing data across the cluster.

Virtualize All the Things … Including the Network!

Server and storage virtualization are very familiar concepts to most, but network virtualization is a relatively new idea. DreamCompute was built from the ground up to provide full network virtualization for every customer. In DreamCompute, the physical network represents an “underlay,” which is invisible to the customer. A virtual network fabric – an “overlay” – is then layered on top, providing every customer in DreamCompute with a virtual OSI Layer 2 (L2) switch, which is completely isolated at L2 from every other customer.

On top of this virtual L2 network, tenants are provided with a virtualized software router, which provides L3+ services like routing, firewalling, and more. DreamHost has open-sourced this project, named it Akanda, and published it under a liberal open source license on GitHub.

DreamCompute is also built from the ground-up to support IPv6 as the exhaustion of IPv4 address space is nearly upon us. Every virtual machine in DreamCompute is automatically assigned an IPv6 address along with its private IPv4 address.

By connecting network virtualization technology with OpenStack’s Neutron Networking APIs, customers have fully programmable control of their network from L2-L7, with isolation.

The Future of the Open Source Cloud is Bright

DreamCompute represents the continuation of a long partnership between DreamHost and the open source community. We’re excited to further our contributions to OpenStack, and to be part of a vibrant ecosystem of cloud service providers who provide OpenStack-based services. The future of the open source cloud is very bright, and we’re delighted to be on the forefront.

DreamHost’s DreamCompute is currently in private beta. To register your interest in joining the free beta period, visit DreamCompute and register today.