Discussion Forums

A "better setiCloud"

22 replies [Last post]
Anders Feder
Offline
Joined: 2010-04-22
Posts: 618

First, a couple of thoughts on how we would like to use AWS:
- We want to run Cloudant's map-reduce service on an instance that we can manage and debug.
- Sigblips want to execute his Baudline analyses on a 'raw' instance.
- The future 'infrastructure group' wants to run the sandbox and other server assets in the cloud.
- I want to enable people to run analyses of the live stream in cloud.

All these uses calls for an environment that makes it easier to manage our AWS account. Currently, it's quite a hassle to go through a SETI employee manually to have them load up a new instance.

I therefore want to propose the following: a web interface that 1) allows anyone to request one or more instances for any number of cpu-hours in the AWS cloud, and 2) allows SETI employees to grant or reject the request with the click of a button - the system should take care of loading up the instances itself.

All of this should be possible using AWS's REST APIs. I volunteer to create a web interface/service like the described.

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
A note on resource usage: It

A note on resource usage:

It would be relevant to have an "Acceptable Use Policy".

However, Amazon also offers something called CloudWatch which monitors resource usage and allows you to program the system to e.g. shut instances down if usages go over a pre-defined quota. CloudWatch is not free to normal users, but can't imagine that Amazon has significant costs associated with it, so maybe they will be willing to let us use it for free.

sigblips
sigblips's picture
Offline
Joined: 2010-04-20
Posts: 733
This looks like a very useful

This looks like a very useful reservation management tool for AWS. I can't wait to be able to use it. Requesting the number of EC2 cpu-hours is a nice touch that should help limit abuse.

Have you seen the AWS command-line-tools?  You can find them by doing a "sudo yum search aws"  I have no idea how they work but everybody had to use them before Amazon built its web-based console dashboard.

.
Here are some thoughts for the Acceptable Use Policy:

* A project name and description and project/scientific goals to help the operator judge the merit of the request.

* No spamming, DDoS'ing, spoofing, other bad stuff, ...

* Zero expectation of privacy.  The SETI Institute reserves the right to login and look around. This will be a huge deterrence to the bad stuff I mentioned above.

* Wasting CPU cycles isn't that big of a deal because Amazon charges by the hour even if the CPU is idle.  Requested CPU-hours is what is important.  All reservations should be setiQuest related but it's probably not worth focusing on CPU usage abuse.

* A project completion report should be written and posted somewhere. I'm thinking something like a simple lab report or a notebook entry. "Description, procedure, results, conclusion." This way the SETI Institute gets something back for letting you use a resource.  A stack of volunteer scientific reports would have a positive effect on setiQuest's public image. Other people might find these reports useful too.  The completion report would also be helpful to the SETI Institute for determining if future resources should be granted to that user.

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
Science reports are a good

Science reports are a good idea, but I don't think it should be a firm requirement - useful services can be envisioned that don't in themselves produce measurable science results (e.g. the sandbox).  But in the request people would have to satisfy SI that the resources will be put to good use.

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
Update

I've installed the AWS SDK for PHP on a new instance and succesfully connected to my own AWS account with it. This simple example just dumps information about availability zones: http://ec2-184-73-59-128.compute-1.amazonaws.com/cloudreserve/api/test.php

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
Identity

Rob has created an user 'identity' for me under the main setiQuest account: http://aws.amazon.com/iam/

I now need Avinash to approve of Rob associating management rights with this user identity, so it can start and stop instances on behalf of the service.

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
I've succesfully connected to

I've succesfully connected to the setiQuest AWS account via the SDK.

Now I am working on the reservation system itself, i.e. the database.

The code tree I am working on will be available here: https://github.com/andersfeder/setiquest-cloudreserve

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
Here is how the UI will look

Here is how the request UI will look to begin with: http://ec2-184-73-59-128.compute-1.amazonaws.com/cloudreserve/request.php

(Completely unstyled, but this can be rectified later on.)

sigblips
sigblips's picture
Offline
Joined: 2010-04-20
Posts: 733
OK, looks good, here are my

OK, looks good, here are my thoughts:

* Missing region and zone controls. These controls are useful for eliminating regional transfer charges and reducing latency. I've done some latency testing and there are other ramifications too. I would like to discuss "regions and zones" at the next IRC meeting. The setiQuest project should select an "official" region and zone but there are reasons why this won't be so simple. Best for these details to be in a different thread.

* The VM image ID should probably default to the basic Amazon or SuSE 64-bit AMI. Just curious, what is ami-3ac33653? The only info I can find about it is the manifest text of amzn-ami-0.9.8-beta.i386-ebs.

* Reservation time of "hours" would make more sense since Amazon charges by the hour. It is also easier for people to do math in their head for hours as apposed to minutes.

* An indefinite reservation time makes a lot of sense.  It probably shouldn't be the default. It might be a good policy to have a text note that says something like "All indefinite duration reservation requests will be denied without prior authorization" but this is really the SETI Institute's call since it's their resource and I don't know what sort of restrictions / quotas apply.

* EBS, S3, RDS, ... reservations. It might be too complicated to setup and control these various AWS services from this sort of interface. I'm not sure how much I would use them and I would understand if they are out of scope of this reservation tool.  Also the user might be able to set these up with the command line tools, need to check.  They might also be enough of a special case to require manual setup with prior authorization.

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
Thanks for the feedback. -

Thanks for the feedback.

- Zone/regions. Yes, my idea was to use a common "official" setting for all requests. I'm interested to hear your arguments against it.
- VM image ID. The 'ami-3ac33653' image is Amazon Linux 32-bit. It would make sense to shift to 64-bit. I'll leave it up to SI.
- Reservation time minutes/hours. My first thought was to use hours. But it occured to me that minutes induce less waste - not just due to granularity but also because a value in minutes psychologically looks bigger. I'll leave it up to SI too, though.
- Indefinite reservation time. I agree there should be a note against it. My intention is that the whole UI will be documented, probably on a seperate page. The note would go there.
- EBS, S3, RDS, ... reservations. Yes, I generally consider that outside the scope of this particular tool. However, I will explore whether it makes sense to make it possible to save an instance to EBS.

sigblips
sigblips's picture
Offline
Joined: 2010-04-20
Posts: 733
> "- VM image ID. The

> "- VM image ID. The 'ami-3ac33653' image is Amazon Linux 32-bit. It would make sense to shift to 64-bit. I'll leave it up to SI."

I just surveyed the Amazon 32-bit AMI's in { us-west, us-east, Ireland, Singapore } and they all have different ID's and none of them match 'ami-3ac33653'.  This leads me to believe that the AMI images are regional and they change regularly (possibly due to kernel and other security updates).

Just checked, the only instances that run with a 32-bit AMI are micro, small. and high-CPU medium instances.  The rest require a 64-bit AMI.  I can't launch the cluster instances so I suspect they require a special AMI or their own.

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
I used the AMI ID listed
sigblips
sigblips's picture
Offline
Joined: 2010-04-20
Posts: 733
It would be interesting what

It would be interesting what kernel version "uname -r" reports for ami-­3ac3365.  The Amazon 64-bit image I launched two days ago reports "2.6.34.7-56.40.amzn1.x86_64".

avinash
Offline
Joined: 2010-01-26
Posts: 278
I finally tried the

I finally tried the system.

  1. Is this system live? I did get to the UI using one of the links, but did not get the email it promised. What am I doing wrong? I am also not sure if that is the latest one.
  2. There are a few mentions of SETI Institute staff approving requests. Is that approval going to be email approval?
  3. There is reference to me assigning usage rights. I am not sure where to do it.

Can you help? Thanks.

Avinash

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
1. The system is under active

[no-glossary]1. The system is live but unfinished and under active development (as in I am tinkering with the code as we speak). There is currently a problem with mails not penetrating spam filters - I am looking into it.

2. The process is as follows: 1) a user files a request for instances, 2) the user receives a confirmation e-mail, 3) the user clicks a link in the confirmation e-mail to verify that he filed the request, 4) the request is sent for approval to an administrator by e-mail, 5) the administrator clicks a link in the e-mail to approve the request, 6) the system automatically launches the needed instances when the reserved time slot begins, and notifies the user, 7) the user can log into the instance(s) and use them as necessary, 8) when the time slot ends, the system kills the instances automatically.

3. Rob has taken care of this, but thanks anyway![/no-glossary]

sigblips
sigblips's picture
Offline
Joined: 2010-04-20
Posts: 733
Is there a way to terminate

Is there a way to terminate an instance reservation early?  Or will "sudo shutdown -h now" do the trick?

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
There will be an interface to

There will be an interface to launch, start, stop and terminate instances on your reservation. When an instance is stopped or terminated this way it will cease expending cpu-minutes on your reservation. Your reservation only becomes unusable once a) all cpu-minutes have been expended or b) it expires (maybe by a fixed expiry date, say one month from being granted).

If you shut the instance down from the command line, it will continue expending cpu-minutes until it is terminated through the web interface (since I can't register the former kind of shutdown).

sigblips
sigblips's picture
Offline
Joined: 2010-04-20
Posts: 733
"- Zone/regions. Yes, my idea

"- Zone/regions. Yes, my idea was to use a common "official" setting for all requests. I'm interested to hear your arguments against it."

I have a desire to request reservation requests for us-east zone-d and us-west. This isn't possible with a single "official" default.

Sidenote: It would be nice if the SETI Institute decided on an official AWS region & zone plan but that might not happen anytime soon or at all.  There are numerous valid reasons why they might want a distributed region and zone plan.

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
The request form now

The request form now validates input: http://ec2-184-73-59-128.compute-1.amazonaws.com/cloudreserve/request.php

You are encouraged to test it (i.e. find invalid values that validates and valid values that doesn't).

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
The request form now adds

The request form now adds validated requests to the database and sends an e-mail back to the user for [no-glossary]confirmation[/no-glossary]: http://ec2-184-72-154-92.compute-1.amazonaws.com/cloudreserve/request.php

A 'boot scripts' field has also been added that will make unattended runs possible - just file your request for an analysis, and let the cloud take care of the rest.

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
Confirmed requests are now

Confirmed requests are now sent by e-mail to an administrator for approval.

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
Due to a number of

Due to a number of complications with sending out e-mail from inside AWS, I had to set up an ad-hoc 'mailer service' on a remote webhost. Maybe someone else can find a better solution.

The spam filter issue should be fixed now, however. Anyone who attempted to use the service but never received an e-mail from it may try again. (Requests still don't actually do anything, yet, though - it's just a test of the database.)

Anders Feder
Offline
Joined: 2010-04-22
Posts: 618
The system now launches

The system now launches instances, woot. Only problem is that it can't stop them again xD

Specifically, the scheduling part (launching when reservation begins, killing when it ends) I still have to figure out how to implement well. Cron/at are suboptimal for web environments.