A note about Aeolus’s direction

Posted on May 16, 2013 by Matt Wagner

Hello all,

I wanted to send an update about the lack of recent activity here. Sadly, most of the core contributors have recently moved onto other endeavors.

One of the truly great things about open source, though, is that this doesn’t have to mean that this is where Aeolus ends. Those of us who have carried the torch thus far are moving on, but the project is still out there. The code is available, and I don’t think that any of us intend to abandon the mailing list or IRC channels in the near future, so we’ll likely be able to help others wanting to pick up where we left off.

The Deltacloud project has just released a similar announcement — the project isn’t going away, but some of the regular contributors are moving on.

Imagefactory and Oz are expected to continue as they exist today and not be impacted. Many other projects, however, will likely not see future releases from the existing set of contributors — including Conductor, Audrey, Tim, aeolus-configure or dev-tools, and Winged Monkey.

Cloud Broker

Posted on April 5, 2013 by Tomas Sedovic

One of the concepts we’ve been thinking about when we started Aeolus was that of a cloud broker. We want to elaborate on what exactly we mean by that and how it compares against other concepts or services in the virtual infrastructure space.

We should start by considering individual infrastructure providers (such as OpenStack or Amazon EC2). Each has its own set of features, a distinct API and a graphical management interface.

Even if everything is open and well documented, there is a de facto lock-in mechanism in place. Migrating your infrastructure from EC2 to Google Compute Engine is far from trivial. Nor is it easy to move from OpenStack to oVirt, despite both being Free and Open Source.

To deal with this, one can use cloud abstraction tools such as Deltacloud, Fog or libcloud. All of these systems provide a one to one mapping to the back end provider, however.

If you want more users to access the same resources, you must share the credentials. You cannot easily enforce quotas and permissions. Talking to multiple providers is an explicit action, too.

This is where the broker comes in.

From the infrastructure user’s perspective, the broker looks and feels just like any other cloud provider: you configure and launch instances, you monitor their status and you shut them down when no longer needed.

The difference between a broker and an abstraction library/service is that the broker can make some decisions before passing the request to the back end provider.

The provider credentials are stored with the broker and the end users don’t have access to them. Instead, they authenticate to the broker. This allows us to enforce quotas and permissions and manage users independently on the actual cloud provider. Moreover, you could hook this to your existing authentication system, such as LDAP.

This allows you to assign a single cloud account to a team or a department and enforce access on your end rather than rely on the provider’s user management capabilities.

With the broker in place, you no longer need a 1-1 mapping between the API and the provider. Transparent to the user, their individual requests may be passed to different providers, based on cost or historical reliability.

By pooling providers and accounts together, you can define your own environments, such as staging on your internal OpenStack setup and production on EC2. The users can work with these environments without having to deal with the underlying infrastructure.

Which means that the broker is an infrastructure provider middleware. It adds business logic between the user and the provider, but the users would not care whether they’re talking to a provider directly or via a broker.

Broker features

Here are some of the features that brokers add on top of vanilla cloud providers:

account multiplexing
credentials
pooling
user management
quota
provider selection (by cost, reliability, etc.)

Broker is not

a UI. While it’s possible to build a user interface on top of a broker, this would be no different from building a UI on top of any cloud provider and thus outside of the broker scope.

multi-instance app manager. A cloud broker deals with the resources defined by the underlying providers: instances, images, flavors/instance types, hardware profiles, networks, etc.

Users interested in app management should use tools such as Heat or CloudFormation. If the tool supports configurable backend providers, the broker could serve as one such backend.

Configuration API

While the broker presents itself as a normal provider, the business logic that it provides must be set up somehow.

Therefore, apart from the user-facing API, the broker needs a separate configuration/management API that allows you to set up the credentials, pools, users, etc.

This API would be available to the people administering the broker but not to the infrastructure end users.

Conclusion

We believe that cloud brokering is a useful service in its own right. It may be interesting to people who don’t want to deal with cloud app management and easier to set up and contribute to.

By behaving as a cloud provider, it could be used in conjunction with app management services such as Heat to provide full Conductor functionality.

We are considering to use the Aeolus Conductor code and experience to build a good Free and Open Source cloud broker. If you’re interested in helping to shape the project, please join the aeolus-devel mailing list where the technical discussion is happening.

OpenStack Heat

Posted on April 3, 2013 by Tomas Sedovic

On and off for the last several months I’ve been involved with another project that may be of interest to the Aeolus community: Heat. It is a project that provides orchestration capabilities to OpenStack.

Heat in a nutshell

Heat was originally modeled after AWS CloudFormation, aiming to bring similar features to OpenStack. It is a service running alongside other OpenStack services such as Nova or Glance and people use it via a ReSTful API or the command line or web clients available.

You start by describing your cloud deployment in a template (which is a file written in a declarative language based on JSON). You specify how many instances instances you want to launch, what images and instance types (small, large) you want to use, what sort of packages and services are to be installed, any files you want to upload, and a host of other possibilities.

Heat will process this template, resolve any dependencies (e.g. the application server needs to be connected to the database server — so it starts the DB first, gets its IP address and passes it to the app server) create and launch the requested resources and make sure they’re in the right state as described in the template.

For example, if you wanted to run a larger MediaWiki setup, you would write a template that has a couple of database servers in different availability zones, have them in a virtual subnet that’s not accessible from the internet, then a few PHP servers running Apache and MediaWiki connected to the databases and a load balancer in front of them for good measure.

Apart from actually launching this cluster, Heat is then able to monitor it (with service and instance granularity) and react to e.g. an increased load by launching a new web server and shutting it down when the load goes down.

About the project

Heat is written in Python and it follows the same architecture as other OpenStack projects. The code is licensed under Apache 2.0 and it’s hosted on GitHub under the OpenStack organisation.

It started as a standalone project but it got recently incubated and it will ship as part of the OpenStack Havana release.

Sounds familiar

You may have noticed there is a bit of an overlap between Aeolus (particularly Conductor and Audrey) and Heat. Specifically, they both support launching and subsequent management of multi-instance deployments.

The main difference is Heat’s current focus on OpenStack as the primary back end and Aeolus’ hard-to-spell name.

Plus the user and quota management features, support for multiple cloud providers, image building, and all the other features Aeolus provides.

So what’s this have to do with Aeolus?

It has been a point of struggle for us to combine all the features we believe are missing in this brave new world of clouds into a single focused goal to guide us forward.

When asked what exactly does Aeolus do, it is all too easy to answer “well, everything, really.”

This shows up in our code base as well.

So here’s a project dedicated to doing one thing (multi-instance app management) well. If we could leverage that work instead of duplicating it, we could focus on doing the other things better without having to deal with all the complexities in our UIs and code.

While Heat is mostly focused towards OpenStack, they do allow different backends via third-party plugins. I wrote a proof of concept plugin that allows Heat to talk to Deltacloud and it seems to work fine. And the Heat community seems to be open to working with other backends even if it’s not their primary goal.

Conclusion

At this point, we’re kicking around ideas on making Aeolus more focused and easier to work with and contribute to. Delegating the Deployment management to Heat and focus on the other features (API, quotas, multiple providers, account multiplexing, etc.) seems like a good step towards that goal.

We’ll share a few of our ideas in the coming days here on the blog and on our mailing list, where the technical discussion is going to happen.

If you’d like to help us shape the future of Aeolus do join the mailing list and participate in the discussion. We are interested in your current and future use cases and any technical feedback.

Where to learn more

If you’re interested in Heat, you can go to the Heat home page, look at the sample templates or go to the #heat IRC channel on Freenode and talk to the users and developers there. They’re very helpful.

You can also check out the AWS CloudFormation documentation page (Heat is compatible with it) and to see Heat in action, you can look at the autoscaling demo screen cast.

Quantum API support for the ruby-openstack rubygem

Posted on March 20, 2013 by marios log

Quantum API support for the ruby-openstack rubygem

As part of work to include Network related resource management the Deltacloud API I added support for the OpenStack Quantum API to the ruby-openstack rubygem. This code will be part of the next release of the rubygem - 1.1.0. For now you need to build the gem from my fork of ruby-openstack and these are some notes for how to try it out:

Build the Rubygem:

Clone from my fork of the repo:

[marios@name Downloads]$ git clone git://github.com/marios/ruby-openstack.git
Cloning into 'ruby-openstack'...
remote: Counting objects: 295, done.
remote: Compressing objects: 100% (155/155), done.
remote: Total 295 (delta 133), reused 282 (delta 126)
Receiving objects: 100% (295/295), 75.52 KiB | 101 KiB/s, done.
Resolving deltas: 100% (133/133), done.

Build the rubygem:

[marios@name Downloads]$ cd ruby-openstack/
[marios@name ruby-openstack]$ gem build openstack.gemspec
Successfully built RubyGem
Name: openstack
Version: 1.1.0
File: openstack-1.1.0.gem

And install:

[marios@name ruby-openstack]$ gem install openstack-1.1.0.gem
Successfully installed openstack-1.1.0
1 gem installed
Installing ri documentation for openstack-1.1.0...
Building YARD (yri) index for openstack-1.1.0...
Installing RDoc documentation for openstack-1.1.0...

Try it out against your Quantum setup:

Using irb, authenticate and get a handle to your Openstack setup. See the README for a full explanation of the parameters - note the use of “network” for “service_type”, specifying that you want a handle to the Quantum service.

[marios@name ~]$ irb -rubygems
irb(main):001:0> require 'openstack'
=> true
irb(main):003:0>  quantum = OpenStack::Connection.create({:username => "admin",
  :api_key=>"le-password", :auth_method=>"password",
  :auth_url => "http://192.168.1.21:5000/v2.0/", :authtenant_name =>"admin",
  :service_type=>"network"})

=> #<OpenStack::Network::Connection:0xa6b0dc8 @connection=#<OpenStack::Connection:0xa6b0f94
    @authuser="admin", @authkey="le-password", @auth_url="http://192.168.1.21:5000/v2.0/",
    @authtenant={:type=>"tenantName", :value=>"admin"}, @auth_method="password",
    @service_name=nil, @service_type="network", @region=nil,
    @regions_list={"RegionOne"=>[{:service=>"image", :versionId=>nil}, {:service=>"compute",
    :versionId=>nil}, {:service=>"ec2", :versionId=>nil}, {:service=>"identity", :versionId=>nil},
     {:service=>"network", :versionId=>nil}]}, @is_debug=nil, @auth_host="192.168.1.21",
    @auth_port=5000, @auth_scheme="http", @auth_path="/v2.0/", @retry_auth=nil, @proxy_host=nil,
    @proxy_port=nil, @authok=true, @http={}, @authtoken="4ac51855458a435089873833b5c225fe",
    @service_host="192.168.1.21", @service_path="/v2.0", @service_port=9696, @service_scheme="http">>

Networks:

Create a Network:

irb(main):016:0> net2 = quantum.create_network("my_net")
=> #<OpenStack::Network::Network:0xa8aff70 @id="6b09a5b3-02d3-4996-9933-4792bd4ca11e",
@name="my_net", @admin_state_up=true, @status="ACTIVE", @subnets=[], @shared=false,
 @tenant_id="7be215d541ea4db4a23b3a84b0882408">

List your Networks:

irb(main):012:0> quantum.networks
=> [#<OpenStack::Network::Network:0xa7c8f94 @id="b4abfaff-9e24-4192-9219-a4a60819aba2",
@name="net_1363343212", @admin_state_up=true, @status="ACTIVE",
@subnets=["34ea9f1e-d71a-48bd-89b0-af2f5c051912",
"7780f890-8334-43a1-934c-f1ce5edc4561"], @shared=false,
@tenant_id="7be215d541ea4db4a23b3a84b0882408">,
#<OpenStack::Network::Network:0xa7c8ef4 @id="1cfad632-3473-4c5b-b2d3-405cc3093286",
@name="net_1363344435", @admin_state_up=true, @status="ACTIVE",
@subnets=["c4293c24-07d0-47dc-bead-ec8d3934b8db"], @shared=false,
@tenant_id="7be215d541ea4db4a23b3a84b0882408">]

irb(main):013:0> net1 = quantum.network("b4abfaff-9e24-4192-9219-a4a60819aba2")
=> #<OpenStack::Network::Network:0xa7f28e4 @id="b4abfaff-9e24-4192-9219-a4a60819aba2",
@name="net_1363343212", @admin_state_up=true, @status="ACTIVE",
@subnets=["34ea9f1e-d71a-48bd-89b0-af2f5c051912",
"7780f890-8334-43a1-934c-f1ce5edc4561"], @shared=false,
@tenant_id="7be215d541ea4db4a23b3a84b0882408">

Delete a Network:

irb(main):017:0> quantum.delete_network("6b09a5b3-02d3-4996-9933-4792bd4ca11e")
=> true

Subnets:

Create a Subnet: - the parameters here are the network_id and cidr

irb(main):030:0> subnet1 = quantum.create_subnet("b4abfaff-9e24-4192-9219-a4a60819aba2",
"192.168.1.0/24")
=> #<OpenStack::Network::Subnet:0xa772a2c @id="e978da1c-9c95-4e58-b4b6-4dcce4640a7f",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @ip_version=4,
@cidr="192.168.1.0/24", @gateway_ip="192.168.1.1", @dns_nameservers=[],
@allocation_pools=[{"start"=>"192.168.1.2", "end"=>"192.168.1.254"}], @host_routes=[],
@enable_dhcp=true, @tenant_id="7be215d541ea4db4a23b3a84b0882408">

List Subnets:

irb(main):014:0> quantum.subnets
=> [#<OpenStack::Network::Subnet:0xa850958 @id="34ea9f1e-d71a-48bd-89b0-af2f5c051912",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @ip_version=4,
@cidr="123.123.0.0/16", @gateway_ip="123.123.0.1", @dns_nameservers=[],
@allocation_pools=[{"start"=>"123.123.0.2", "end"=>"123.123.255.254"}],
@host_routes=[], @enable_dhcp=true, @tenant_id="7be215d541ea4db4a23b3a84b0882408">,
#<OpenStack::Network::Subnet:0xa850868 @id="7780f890-8334-43a1-934c-f1ce5edc4561",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @ip_version=4,
@cidr="192.168.1.0/24", @gateway_ip="192.168.1.1", @dns_nameservers=[],
@allocation_pools=[{"start"=>"192.168.1.2", "end"=>"192.168.1.254"}], @host_routes=[],
@enable_dhcp=true, @tenant_id="7be215d541ea4db4a23b3a84b0882408">]

irb(main):015:0> subnet1 = quantum.subnet("34ea9f1e-d71a-48bd-89b0-af2f5c051912")
=> #<OpenStack::Network::Subnet:0xa88ec1c @id="34ea9f1e-d71a-48bd-89b0-af2f5c051912",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @ip_version=4,
@cidr="123.123.0.0/16", @gateway_ip="123.123.0.1", @dns_nameservers=[],
@allocation_pools=[{"start"=>"123.123.0.2", "end"=>"123.123.255.254"}],
@host_routes=[], @enable_dhcp=true, @tenant_id="7be215d541ea4db4a23b3a84b0882408">

Delete a Subnet:

irb(main):023:0> quantum.delete_subnet("89136e4d-da26-4495-9384-0fd1309e274a")
=> true

Ports:

Create a Port - note that the port is created on the specified Network. Furthermore, the “fixed_ips” structure specified the subnet for the port, as well as the’device-id’ which associates that server with this port (and network+subnet):

irb(main):058:0> created_port = quantum.create_port(network.id,
{"fixed_ips"=>[{"subnet_id"=>subnet.id}], "device_id"=>machine.id})

=> #<OpenStack::Network::Port:0x95a833c @id="d601db9e-c936-4811-904a-bb5a27d105f3",
@network_id="c4dfe90e-a7ce-41f7-b9b2-2f9773f42a6b", @name="", @admin_state_up=true,
@status="ACTIVE", @mac_address="fa:16:3e:f4:e8:bc",
@fixed_ips=[{"subnet_id"=>"f78bfc05-ead0-40a6-8325-a35eb2b535c3", "ip_address"=>"10.0.0.4"}],
@device_id="fe4022fa-a77c-4adf-be45-6e069fb3a314", @device_owner="",
@tenant_id="7be215d541ea4db4a23b3a84b0882408">

List all Ports:

irb(main):032:0> quantum.ports
=> [#<OpenStack::Network::Port:0xa73bdd8 @id="f0db2c95-9449-4eb3-8b12-c66b23c8de41",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @admin_state_up=true,
@status="ACTIVE", @mac_address="fa:16:3e:7e:e0:1b",
@fixed_ips=[{"subnet_id"=>"34ea9f1e-d71a-48bd-89b0-af2f5c051912",
"ip_address"=>"123.123.0.2"}], @device_id="4357665c-e2fe-4bf0-8a69-b531d11add2d",
@device_owner="compute:nova", @tenant_id="7be215d541ea4db4a23b3a84b0882408">,
#<OpenStack::Network::Port:0xa73bcac @id="93c210cd-b086-49e1-aa12-d207aa28a981",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @admin_state_up=true,
@status="ACTIVE", @mac_address="fa:16:3e:08:eb:92",
@fixed_ips=[{"subnet_id"=>"34ea9f1e-d71a-48bd-89b0-af2f5c051912",
"ip_address"=>"123.123.0.3"}], @device_id="5a114ba2-4f49-4655-93f9-96954632cb67",
@device_owner="compute:nova", @tenant_id="7be215d541ea4db4a23b3a84b0882408">]


irb(main):033:0> port1 = quantum.port("f0db2c95-9449-4eb3-8b12-c66b23c8de41")
=> #<OpenStack::Network::Port:0xa70f51c @id="f0db2c95-9449-4eb3-8b12-c66b23c8de41",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @admin_state_up=true,
@status="ACTIVE", @mac_address="fa:16:3e:7e:e0:1b",
@fixed_ips=[{"subnet_id"=>"34ea9f1e-d71a-48bd-89b0-af2f5c051912",
"ip_address"=>"123.123.0.2"}], @device_id="4357665c-e2fe-4bf0-8a69-b531d11add2d",
@device_owner="compute:nova", @tenant_id="7be215d541ea4db4a23b3a84b0882408">

Delete a Port:

irb(main):034:0> quantum.delete_port("f0db2c95-9449-4eb3-8b12-c66b23c8de41")
=> true

Quantum API support for the ruby-openstack rubygem

Posted on March 20, 2013 by marios log

Quantum API support for the ruby-openstack rubygem

Build the Rubygem:

Clone from my fork of the repo:

[marios@name Downloads]$ git clone git://github.com/marios/ruby-openstack.git
Cloning into 'ruby-openstack'...
remote: Counting objects: 295, done.
remote: Compressing objects: 100% (155/155), done.
remote: Total 295 (delta 133), reused 282 (delta 126)
Receiving objects: 100% (295/295), 75.52 KiB | 101 KiB/s, done.
Resolving deltas: 100% (133/133), done.

Build the rubygem:

[marios@name Downloads]$ cd ruby-openstack/
[marios@name ruby-openstack]$ gem build openstack.gemspec
Successfully built RubyGem
Name: openstack
Version: 1.1.0
File: openstack-1.1.0.gem

And install:

[marios@name ruby-openstack]$ gem install openstack-1.1.0.gem
Successfully installed openstack-1.1.0
1 gem installed
Installing ri documentation for openstack-1.1.0...
Building YARD (yri) index for openstack-1.1.0...
Installing RDoc documentation for openstack-1.1.0...

Try it out against your Quantum setup:

[marios@name ~]$ irb -rubygems
irb(main):001:0> require 'openstack'
=> true
irb(main):003:0>  quantum = OpenStack::Connection.create({:username => "admin",
  :api_key=>"le-password", :auth_method=>"password",
  :auth_url => "http://192.168.1.21:5000/v2.0/", :authtenant_name =>"admin",
  :service_type=>"network"})

=> #<OpenStack::Network::Connection:0xa6b0dc8 @connection=#<OpenStack::Connection:0xa6b0f94
    @authuser="admin", @authkey="le-password", @auth_url="http://192.168.1.21:5000/v2.0/",
    @authtenant={:type=>"tenantName", :value=>"admin"}, @auth_method="password",
    @service_name=nil, @service_type="network", @region=nil,
    @regions_list={"RegionOne"=>[{:service=>"image", :versionId=>nil}, {:service=>"compute",
    :versionId=>nil}, {:service=>"ec2", :versionId=>nil}, {:service=>"identity", :versionId=>nil},
     {:service=>"network", :versionId=>nil}]}, @is_debug=nil, @auth_host="192.168.1.21",
    @auth_port=5000, @auth_scheme="http", @auth_path="/v2.0/", @retry_auth=nil, @proxy_host=nil,
    @proxy_port=nil, @authok=true, @http={}, @authtoken="4ac51855458a435089873833b5c225fe",
    @service_host="192.168.1.21", @service_path="/v2.0", @service_port=9696, @service_scheme="http">>

Networks:

Create a Network:

irb(main):016:0> net2 = quantum.create_network("my_net")
=> #<OpenStack::Network::Network:0xa8aff70 @id="6b09a5b3-02d3-4996-9933-4792bd4ca11e",
@name="my_net", @admin_state_up=true, @status="ACTIVE", @subnets=[], @shared=false,
 @tenant_id="7be215d541ea4db4a23b3a84b0882408">

List your Networks:

irb(main):012:0> quantum.networks
=> [#<OpenStack::Network::Network:0xa7c8f94 @id="b4abfaff-9e24-4192-9219-a4a60819aba2",
@name="net_1363343212", @admin_state_up=true, @status="ACTIVE",
@subnets=["34ea9f1e-d71a-48bd-89b0-af2f5c051912",
"7780f890-8334-43a1-934c-f1ce5edc4561"], @shared=false,
@tenant_id="7be215d541ea4db4a23b3a84b0882408">,
#<OpenStack::Network::Network:0xa7c8ef4 @id="1cfad632-3473-4c5b-b2d3-405cc3093286",
@name="net_1363344435", @admin_state_up=true, @status="ACTIVE",
@subnets=["c4293c24-07d0-47dc-bead-ec8d3934b8db"], @shared=false,
@tenant_id="7be215d541ea4db4a23b3a84b0882408">]

irb(main):013:0> net1 = quantum.network("b4abfaff-9e24-4192-9219-a4a60819aba2")
=> #<OpenStack::Network::Network:0xa7f28e4 @id="b4abfaff-9e24-4192-9219-a4a60819aba2",
@name="net_1363343212", @admin_state_up=true, @status="ACTIVE",
@subnets=["34ea9f1e-d71a-48bd-89b0-af2f5c051912",
"7780f890-8334-43a1-934c-f1ce5edc4561"], @shared=false,
@tenant_id="7be215d541ea4db4a23b3a84b0882408">

Delete a Network:

irb(main):017:0> quantum.delete_network("6b09a5b3-02d3-4996-9933-4792bd4ca11e")
=> true

Subnets:

Create a Subnet: - the parameters here are the network_id and cidr

irb(main):030:0> subnet1 = quantum.create_subnet("b4abfaff-9e24-4192-9219-a4a60819aba2",
"192.168.1.0/24")
=> #<OpenStack::Network::Subnet:0xa772a2c @id="e978da1c-9c95-4e58-b4b6-4dcce4640a7f",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @ip_version=4,
@cidr="192.168.1.0/24", @gateway_ip="192.168.1.1", @dns_nameservers=[],
@allocation_pools=[{"start"=>"192.168.1.2", "end"=>"192.168.1.254"}], @host_routes=[],
@enable_dhcp=true, @tenant_id="7be215d541ea4db4a23b3a84b0882408">

List Subnets:

irb(main):014:0> quantum.subnets
=> [#<OpenStack::Network::Subnet:0xa850958 @id="34ea9f1e-d71a-48bd-89b0-af2f5c051912",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @ip_version=4,
@cidr="123.123.0.0/16", @gateway_ip="123.123.0.1", @dns_nameservers=[],
@allocation_pools=[{"start"=>"123.123.0.2", "end"=>"123.123.255.254"}],
@host_routes=[], @enable_dhcp=true, @tenant_id="7be215d541ea4db4a23b3a84b0882408">,
#<OpenStack::Network::Subnet:0xa850868 @id="7780f890-8334-43a1-934c-f1ce5edc4561",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @ip_version=4,
@cidr="192.168.1.0/24", @gateway_ip="192.168.1.1", @dns_nameservers=[],
@allocation_pools=[{"start"=>"192.168.1.2", "end"=>"192.168.1.254"}], @host_routes=[],
@enable_dhcp=true, @tenant_id="7be215d541ea4db4a23b3a84b0882408">]

irb(main):015:0> subnet1 = quantum.subnet("34ea9f1e-d71a-48bd-89b0-af2f5c051912")
=> #<OpenStack::Network::Subnet:0xa88ec1c @id="34ea9f1e-d71a-48bd-89b0-af2f5c051912",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @ip_version=4,
@cidr="123.123.0.0/16", @gateway_ip="123.123.0.1", @dns_nameservers=[],
@allocation_pools=[{"start"=>"123.123.0.2", "end"=>"123.123.255.254"}],
@host_routes=[], @enable_dhcp=true, @tenant_id="7be215d541ea4db4a23b3a84b0882408">

Delete a Subnet:

irb(main):023:0> quantum.delete_subnet("89136e4d-da26-4495-9384-0fd1309e274a")
=> true

Ports:

irb(main):058:0> created_port = quantum.create_port(network.id,
{"fixed_ips"=>[{"subnet_id"=>subnet.id}], "device_id"=>machine.id})

=> #<OpenStack::Network::Port:0x95a833c @id="d601db9e-c936-4811-904a-bb5a27d105f3",
@network_id="c4dfe90e-a7ce-41f7-b9b2-2f9773f42a6b", @name="", @admin_state_up=true,
@status="ACTIVE", @mac_address="fa:16:3e:f4:e8:bc",
@fixed_ips=[{"subnet_id"=>"f78bfc05-ead0-40a6-8325-a35eb2b535c3", "ip_address"=>"10.0.0.4"}],
@device_id="fe4022fa-a77c-4adf-be45-6e069fb3a314", @device_owner="",
@tenant_id="7be215d541ea4db4a23b3a84b0882408">

List all Ports:

irb(main):032:0> quantum.ports
=> [#<OpenStack::Network::Port:0xa73bdd8 @id="f0db2c95-9449-4eb3-8b12-c66b23c8de41",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @admin_state_up=true,
@status="ACTIVE", @mac_address="fa:16:3e:7e:e0:1b",
@fixed_ips=[{"subnet_id"=>"34ea9f1e-d71a-48bd-89b0-af2f5c051912",
"ip_address"=>"123.123.0.2"}], @device_id="4357665c-e2fe-4bf0-8a69-b531d11add2d",
@device_owner="compute:nova", @tenant_id="7be215d541ea4db4a23b3a84b0882408">,
#<OpenStack::Network::Port:0xa73bcac @id="93c210cd-b086-49e1-aa12-d207aa28a981",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @admin_state_up=true,
@status="ACTIVE", @mac_address="fa:16:3e:08:eb:92",
@fixed_ips=[{"subnet_id"=>"34ea9f1e-d71a-48bd-89b0-af2f5c051912",
"ip_address"=>"123.123.0.3"}], @device_id="5a114ba2-4f49-4655-93f9-96954632cb67",
@device_owner="compute:nova", @tenant_id="7be215d541ea4db4a23b3a84b0882408">]


irb(main):033:0> port1 = quantum.port("f0db2c95-9449-4eb3-8b12-c66b23c8de41")
=> #<OpenStack::Network::Port:0xa70f51c @id="f0db2c95-9449-4eb3-8b12-c66b23c8de41",
@network_id="b4abfaff-9e24-4192-9219-a4a60819aba2", @name="", @admin_state_up=true,
@status="ACTIVE", @mac_address="fa:16:3e:7e:e0:1b",
@fixed_ips=[{"subnet_id"=>"34ea9f1e-d71a-48bd-89b0-af2f5c051912",
"ip_address"=>"123.123.0.2"}], @device_id="4357665c-e2fe-4bf0-8a69-b531d11add2d",
@device_owner="compute:nova", @tenant_id="7be215d541ea4db4a23b3a84b0882408">

Delete a Port:

irb(main):034:0> quantum.delete_port("f0db2c95-9449-4eb3-8b12-c66b23c8de41")
=> true

Deltacloud Networks API – a work in progress

Posted on March 19, 2013 by marios log

Deltacloud Networks API - a work in progress

The Deltacloud API is missing the management of network resources. Many IaaS APIs still don’t have a notion of ‘network’ resource management whilst for others the Network API is relatively new. However, the Deltacoud project has received requests for the addition of network management related functionality and we’ve been discussing this on and off for the past year or so. Towards the end of 2012 we made a first attempt with some network models to fuel discussion and this blog post introduces the second iteration of the code for further comments.

Each member of the ‘networks team’ took ‘ownership’ of a particular cloud provider to discuss how the initial model would ‘fit’ against that provider’s network API. This etherpad has some of our preliminary notes; Jan Provaznik looked at RHEV-M, Michal Fojtik looked at vSphere, Dies Koper looked at FGCP, Francesco Vollero looked at EC2 and I looked at OpenStack and CIMI. Tomas Sedovic acted as UN weapons inspector and sanity checker.

Angus Thomas developed the use cases to guide development. For this first iteration I focused on the simplest use case: “Place an Instance into a particular Network”. Furthermore, I decided to focus on putting down some fully working code for EC2 and OpenStack to explore the feasibility of the Networks abstractions and guide further development for the other drivers.

The Network Models

The original model had three entities - Network, Subnet and Port. The ‘Port’ entity proved to be problematic, because for some cloud providers this represented a ‘switch port’ whereas for others it represented a ‘network interface’. In Deltacloud we just want to model the connection of an Instance to a Network. So I decided to remove the Port model altogether and replace it with an ‘instance_bindings’ attribute on the Instance resource. This attempts to abstract away or reconcile the differences between the two ‘types’ of Port resource found in the various cloud provider APIs.

As I attempt to illustrate above, Network and Subnet remain as stand alone resources implying full CRUD operations for these where supported. The Network and Subnet models are pretty similar with respect to their attributes. One not so obvious distinction worth noting is that Network has an ‘address_blocks’ attribute whereas Subnet has an ‘address_block’ attribute. This is because for Openstack Quantum, the Subnets that are associated with a given Network do not have to utilise/occupy contiguous CIDR address blocks - that is it is perfectly legal for one subnet to use “10.0.0.0/8” and the other to use “192.168.0.0/16”. These two address blocks cannot be described using a single CIDR string. Hence, for Network, address_blocks is an array of CIDR address strings to accommodate for this difference.

The Instance model is extended with the network_bindings attribute - an array - because an Instance can be associated with more than one Network (in some providers). Each network_binding contains a reference to the network and subnet as well as the IP address resulting from this association. Moving forward we can either extend this abstraction to contain more attributes or if necessary and feasible, extract it back into a stand-alone entity.

Common Operations and Driver Patterns

Networks

For EC2 the Network resource is mapped to the EC2 VPC, whilst for Openstack it is mapped to the Quantum Network. In EC2, each Netowrk (VPC) has a ‘cidr’ attribute whereas for Openstack a Quantum network does not (cidr is present on the Quantum Subnet resource). For Openstack, each Network ‘knows’ which subnets belong to it, whereas for EC2 this information must be deduced by filtering the Subnets.

Populating the list of Network resources in the case of EC2:

    def networks(credentials, opts={})
      ec2 = new_client(credentials)
      networks = []
      safely do
        subnets = subnets(credentials) #get all subnets once
        ec2.describe_vpcs.each do |vpc|
          vpc_subnets = subnets.inject([]){|res,cur| res<<cur if cur.network==vpc[:vpc_id]  ;res} #collect subnets for this.network
          networks << convert_vpc(vpc, vpc_subnets)
        end
      end
      networks = filter_on(networks, :id, opts)
    end

    def convert_vpc(vpc, subnets=[])
      addr_blocks = subnets.inject([]){|res,cur| res << cur.address_block  ; res}
      Network.new({ :id => vpc[:vpc_id],
                    :name => vpc[:vpc_id],
                    :state=> vpc[:state],
                    :subnets => subnets.inject([]){|res,cur| res << cur.id  ;res},
                    :address_blocks=> (addr_blocks.empty? ? [vpc[:cidr_block]] : addr_blocks)  })
    end

In other words, you retrieve the list of subnets, then for each VPC returned create a Deltacloud::Network resource using the VPC and the subset of the Subnets that are associated with that. One distinction/point of discussion here is that for the ‘addr_blocks’ attribute (cidr), when there are subnets in the given VPC, I only report the CIDR addresses of those subnets in address_blocks. The alternative is to simply use the :cidr_block of the entire VPC, but it made more sense to me to report only the addresses that an Instance can actually use at a given time.

For Openstack however, each Network resource ‘knows’ which subnets belong to it. However, a Network does not ‘know’ about the CIDR address blocks in use on those subnets. Hence:

    def networks(credentials, opts={})
      os = new_client(credentials, "network")
      networks = []
      safely do
        subnets = os.subnets
        os.networks.each do |net|
          addr_blocks = get_address_blocks_for(net.id, subnets)
          networks << convert_network(net, addr_blocks)
        end
      end

    def get_address_blocks_for(network_id, subnets)
      return [] if subnets.empty?
      addr_blocks = []
      subnets.each do |sn|
        if sn.network_id == network_id
          addr_blocks << sn.cidr
        end
      end
      addr_blocks
    end

    def convert_network(net, addr_blocks)
      Network.new({ :id => net.id,
                    :name => net.name,
                    :subnets => net.subnets,
                    :state => (net.admin_state_up ? "UP" : "DOWN"),
                    :address_blocks => addr_blocks
      })
    end

In other words, you must retrieve the list of subnets to determine the CIDR address blocks for those subnets belonging to the given Network, before converting the result into a Deltacloud::Network.

One final implication of the difference between EC2 VPC ‘knowing’ about CIDR and Openstack Network not knowing, is that creation of a Network in EC2 demands a cidr block. This difference will have to be somehow reconciled - either by making ‘cidr’ a compulsory parameter that is ignored for Openstack, or making it optional and choosing a default for EC2 when it isn’t supplied. We could also advertise the difference using a driver ‘feature’.

Subnets

Listing of subnets is thankfully more straightforward without any big differences between the two providers. Each returned subnet resource contains all the attributes required to build the Deltacloud::Subnet. In both cases, a subnet ‘knows’ about the Network it belongs to as well as the CIDR block it occupies/uses. For the create operation in both cases you must supply network_id and address_block parameters:

    def create_subnet(credentials, opts={})
      ec2 = new_client(credentials)
      safely do
        subnet = ec2.create_subnet(opts[:network_id], opts[:address_block])
        convert_subnet(subnet)
      end
    end

    def convert_subnet(subnet)
      Subnet.new({  :id => subnet[:subnet_id],
                    :name => subnet[:subnet_id],
                    :network =>subnet[:vpc_id],
                    :address_block => subnet[:cidr_block],
                    :state => subnet[:state] })
    end

Instances

The Instance operations impacted for this initial use case are creation, listing (retrieval) as well as deletion. When creating an Instance a client must provide the network_id and subnet_id parameters.

For EC2, placing an Instance into a particular Network during Instance creation is very straightforward, because the EC2 compute and network APIs are exposed via the same interface (the EC2 API). You simply pass the ‘subnet_id’ parameter to the EC2 instance creation operation and extract the required ‘network_bindings’ from the response:

    def convert_instance(instance)
    ...
      if instance[:vpc_id]
        inst_params.merge!(:network_bindings => [{:network=>instance[:vpc_id], :subnet=>instance[:subnet_id], :ip_address=> instance[:aws_private_ip_address]}])
      end
      Instance.new(inst_params)
    end

However, in the case of Openstack, the Quantum and Nova APIs do not have such a straightforward interface. To place an Instance into a particular Network, you must first create the Instance, then create a Quantum Port on the required Network and Subnet and then set the Port to refer to the created Instance:

    def create_instance(credentials, image_id, opts)
      (...)
      safely do
        server = os.create_server(params)
        net_bind = []
        if opts[:network_id] && opts[:subnet_id] && (quantum=have_quantum?(credentials)) #place instance into a network
          port = quantum.create_port(opts[:network_id], {"fixed_ips"=>[{"subnet_id"=>opts[:subnet_id]}], "device_id"=>server.id})
          net_bind = [{:network=>opts[:network], :subnet=>opts[:subnet_id], :ip_address=>port.fixed_ips.first["ip_address"]}]
          server.refresh
        end
        result = convert_from_server(server, os.connection.authuser, get_attachments(server.id, os), net_bind)
      end
      result
    end

For the same reasons (interplay between compute and network APIs), creation of the network_bindings attribute when listing Instance(s) is much simpler for EC2 - you simply extract the required information from the EC2 response. For Openstack however you must deduce the network bindings by retrieving a list of Quantum Ports and discovering which of those refers to the given Instance:

    def instances(credentials, opts={})
      (...)
      insts = attachments = ports = []
      safely do
        if quantum = have_quantum?(credentials)
          ports = quantum.ports
        end
        (...)
          insts = os.list_servers_detail.collect do |s|
            net_bind = get_net_bindings_for(s[:id], ports)
            convert_from_server(s, os.connection.authuser,get_attachments(s[:id], os), net_bind)
          end
        end
      end
      insts = filter_on( insts, :state, opts )
      insts
    end

    def get_net_bindings_for(server_id, ports)
      return [] if ports.empty?
      net_bind = []
      ports.each do |port|
        if port.device_id == server_id
          port.fixed_ips.each do |fix_ip|
            net_bind << {:network=> port.network_id, :subnet=>fix_ip["subnet_id"] , :ip_address=>fix_ip["ip_address"]}
          end
        end
      end
      net_bind
    end

A final note is that for the destroy_instance operation in the Openstack driver I added an extra ‘cleanup’ step to destroy any ports associated with the given Instance:

    def destroy_instance(credentials, instance_id)
      os = new_client(credentials)
      server = instance = nil
      safely do
        server = os.get_server(instance_id)
        server.delete!
        if quantum = have_quantum?(credentials) #destroy ports if any
          quantum.ports.each do |port|
            if port.device_id == server.id
              quantum.delete_port(port.id)
            end
          end
        end
      (...)
    end

back top

Patches

The patches implementing this functionality have been sent to the dev@deltacloud.apache.org mailing list. They are also available from tracker as set 395. One note is that in order to use the Openstack Quantum code you need to build from my fork of ruby-openstack; I’ll be releasing a newer version of the gem (1.10) that includes this functionality in the next few days.

Note that the current patches have incomplete ‘views’ - in particular the XML/JSON views are not fully implemented. I mainly used the HTML interface for testing and so the HTML UI should be used to review this set for now.

Open Issues/Questions

How do these revised models and operations fit for your IaaS API (RHEV-M, vSphere, FGCP, CIMI).

For example, for RHEV-M we know that there is no separation of Network and Subnet. Furthermore, creation of Networks is an admin-level task and so the create_network method likely wouldn’t be exposed for the RHEV-M driver. With respect to placing an Instance into a particular Network I understand the procedure is similar to Openstack Quantum: create an instance and then create a network interface on that instance which is associated with a particular Network. I will likely be targetting the RHEV-M driver next to explore these issues further.

How do we reconcile the fact that in some clouds, there is no ‘subnet’ entity (e.g. RHEV-M).

One implication is that for operations such as ‘create_instance’ where we may mandate that a user must supply both a network_id as well as a subnet_id. We could only require the subnet_id (with the network_id being deduced from that). However, this brings us back to the ‘what if there is no subnet’? Do we then map given IaaS provider’s ‘Network’ resource to the Deltacloud ‘Subnet’ (and only expose ‘Subnet’ but not ‘Network’ at the top-level /api entry point?)?

Do we want to merge the Network and Subnet models?

The logic here is that for all the operations regarding an Instance and indeed the network_binding association of Instance <–> Network+Subnet involves both network_id and subnet_id. In a given IaaS provider that supports both Network and Subnet models, you associate your Instances with a Subnet, not a Network. So if you have 2 networks each containing 3 subnets, in effect you have a choice of 6 subnets to which an Instance can be associated - each Network+Subnet combination becomes one target for the association. This idea needs further exploration/discussion, though it may solve the issue with those providers that don’t support a 2 tier Networking model (like RHEV-M, CIMI).

Deltacloud Networks API – a work in progress

Posted on March 19, 2013 by marios log

Deltacloud Networks API - a work in progress

The Network Models

Common Operations and Driver Patterns

Networks

Populating the list of Network resources in the case of EC2:

    def networks(credentials, opts={})
      ec2 = new_client(credentials)
      networks = []
      safely do
        subnets = subnets(credentials) #get all subnets once
        ec2.describe_vpcs.each do |vpc|
          vpc_subnets = subnets.inject([]){|res,cur| res<<cur if cur.network==vpc[:vpc_id]  ;res} #collect subnets for this.network
          networks << convert_vpc(vpc, vpc_subnets)
        end
      end
      networks = filter_on(networks, :id, opts)
    end

    def convert_vpc(vpc, subnets=[])
      addr_blocks = subnets.inject([]){|res,cur| res << cur.address_block  ; res}
      Network.new({ :id => vpc[:vpc_id],
                    :name => vpc[:vpc_id],
                    :state=> vpc[:state],
                    :subnets => subnets.inject([]){|res,cur| res << cur.id  ;res},
                    :address_blocks=> (addr_blocks.empty? ? [vpc[:cidr_block]] : addr_blocks)  })
    end

For Openstack however, each Network resource ‘knows’ which subnets belong to it. However, a Network does not ‘know’ about the CIDR address blocks in use on those subnets. Hence:

    def networks(credentials, opts={})
      os = new_client(credentials, "network")
      networks = []
      safely do
        subnets = os.subnets
        os.networks.each do |net|
          addr_blocks = get_address_blocks_for(net.id, subnets)
          networks << convert_network(net, addr_blocks)
        end
      end

    def get_address_blocks_for(network_id, subnets)
      return [] if subnets.empty?
      addr_blocks = []
      subnets.each do |sn|
        if sn.network_id == network_id
          addr_blocks << sn.cidr
        end
      end
      addr_blocks
    end

    def convert_network(net, addr_blocks)
      Network.new({ :id => net.id,
                    :name => net.name,
                    :subnets => net.subnets,
                    :state => (net.admin_state_up ? "UP" : "DOWN"),
                    :address_blocks => addr_blocks
      })
    end

In other words, you must retrieve the list of subnets to determine the CIDR address blocks for those subnets belonging to the given Network, before converting the result into a Deltacloud::Network.

Subnets

    def create_subnet(credentials, opts={})
      ec2 = new_client(credentials)
      safely do
        subnet = ec2.create_subnet(opts[:network_id], opts[:address_block])
        convert_subnet(subnet)
      end
    end

    def convert_subnet(subnet)
      Subnet.new({  :id => subnet[:subnet_id],
                    :name => subnet[:subnet_id],
                    :network =>subnet[:vpc_id],
                    :address_block => subnet[:cidr_block],
                    :state => subnet[:state] })
    end

Instances

    def convert_instance(instance)
    ...
      if instance[:vpc_id]
        inst_params.merge!(:network_bindings => [{:network=>instance[:vpc_id], :subnet=>instance[:subnet_id], :ip_address=> instance[:aws_private_ip_address]}])
      end
      Instance.new(inst_params)
    end

    def create_instance(credentials, image_id, opts)
      (...)
      safely do
        server = os.create_server(params)
        net_bind = []
        if opts[:network_id] && opts[:subnet_id] && (quantum=have_quantum?(credentials)) #place instance into a network
          port = quantum.create_port(opts[:network_id], {"fixed_ips"=>[{"subnet_id"=>opts[:subnet_id]}], "device_id"=>server.id})
          net_bind = [{:network=>opts[:network], :subnet=>opts[:subnet_id], :ip_address=>port.fixed_ips.first["ip_address"]}]
          server.refresh
        end
        result = convert_from_server(server, os.connection.authuser, get_attachments(server.id, os), net_bind)
      end
      result
    end

    def instances(credentials, opts={})
      (...)
      insts = attachments = ports = []
      safely do
        if quantum = have_quantum?(credentials)
          ports = quantum.ports
        end
        (...)
          insts = os.list_servers_detail.collect do |s|
            net_bind = get_net_bindings_for(s[:id], ports)
            convert_from_server(s, os.connection.authuser,get_attachments(s[:id], os), net_bind)
          end
        end
      end
      insts = filter_on( insts, :state, opts )
      insts
    end

    def get_net_bindings_for(server_id, ports)
      return [] if ports.empty?
      net_bind = []
      ports.each do |port|
        if port.device_id == server_id
          port.fixed_ips.each do |fix_ip|
            net_bind << {:network=> port.network_id, :subnet=>fix_ip["subnet_id"] , :ip_address=>fix_ip["ip_address"]}
          end
        end
      end
      net_bind
    end

A final note is that for the destroy_instance operation in the Openstack driver I added an extra ‘cleanup’ step to destroy any ports associated with the given Instance:

    def destroy_instance(credentials, instance_id)
      os = new_client(credentials)
      server = instance = nil
      safely do
        server = os.get_server(instance_id)
        server.delete!
        if quantum = have_quantum?(credentials) #destroy ports if any
          quantum.ports.each do |port|
            if port.device_id == server.id
              quantum.delete_port(port.id)
            end
          end
        end
      (...)
    end

back top

Patches

Open Issues/Questions

How do these revised models and operations fit for your IaaS API (RHEV-M, vSphere, FGCP, CIMI).

How do we reconcile the fact that in some clouds, there is no ‘subnet’ entity (e.g. RHEV-M).

Do we want to merge the Network and Subnet models?

Aeolus will be at LOADays, 2013

Posted on March 19, 2013 by tzumainn

Aeolus will be presenting at LOADays! This conference is located in Antwerp, Belgium, and takes place over April 6 – 7, 2013.

Aeolus and Katello FTW (For the Win) - Francesco Vollero

Recent years have seen an influx of Cloud Computing services offered by a plethora of vendors with many different APIs and interfaces. The diversity of these products offer many solutions at various levels of the stack, but increases the risk of proprietary cloud adoption due to vendor lock-in and downtime.

With Aeolus we aim to provide a Free and Open Source cloud computing framework through which instances can be launched against any cloud provider using the same API and toolset. The infrastructure permits the description of software services and hardware components in such a way that any set of software packages can be deployed to any cloud provider matching specified criteria.

For Katello I will show how to manage running instances, apply errata, do updates, compare status with deployed images, manage content effectively. All of this to simplify the life of sysadmins and devops.