Debian Bug report logs - #694035
JSON based API to query available AMIs

Package: cloud.debian.org; Maintainer for cloud.debian.org is Debian Cloud Team <debian-cloud@lists.debian.org>;

Reported by: Charles Plessy <plessy@debian.org>

Date: Fri, 23 Nov 2012 05:39:02 UTC

Severity: wishlist

Reply or subscribe to this bug.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to debian-bugs-dist@lists.debian.org, Debian Cloud Team <debian-cloud@lists.debian.org>:
Bug#694035; Package cloud.debian.org. (Fri, 23 Nov 2012 05:39:05 GMT) Full text and rfc822 format available.

Acknowledgement sent to Charles Plessy <plessy@debian.org>:
New Bug report received and forwarded. Copy sent to Debian Cloud Team <debian-cloud@lists.debian.org>. (Fri, 23 Nov 2012 05:39:05 GMT) Full text and rfc822 format available.

Message #5 received at submit@bugs.debian.org (full text, mbox):

From: Charles Plessy <plessy@debian.org>
To: submit@bugs.debian.org
Subject: JSON based API to query available AMIs
Date: Fri, 23 Nov 2012 14:34:51 +0900
Package: cloud.debian.org
Severity: wishlist

Le Tue, Nov 20, 2012 at 01:28:12PM -0800, Eric Hammond a écrit :
> 
> Here is a new proposed format for returning AMI information in a
> JSON format:
> 
> 
> https://blueprints.launchpad.net/ubuntu/+spec/servercloud-r-query2-deprecation
> 
> A lot of experience releasing AMIs has gone into creating this
> definition, so I'd recommend adopting this standard if you can.

Creating a bug to keep track of Eric's suggestion.

Cheers,

-- 
Charles



Information stored :
Bug#694035; Package cloud.debian.org. (Sat, 24 Nov 2012 10:39:13 GMT) Full text and rfc822 format available.

Acknowledgement sent to Stefano Zacchiroli <zack@debian.org>:
Extra info received and filed, but not forwarded. (Sat, 24 Nov 2012 10:39:13 GMT) Full text and rfc822 format available.

Message #10 received at 694035-quiet@bugs.debian.org (full text, mbox):

From: Stefano Zacchiroli <zack@debian.org>
To: debian-cloud@lists.debian.org
Cc: 694035-quiet@bugs.debian.org
Subject: Re: JSON based API to query available AMIs
Date: Sat, 24 Nov 2012 11:35:58 +0100
[Message part 1 (text/plain, inline)]
[ adding the bug log to Cc, to keep track of relevant information ]

On Thu, Nov 22, 2012 at 10:52:46PM -0800, Eric Hammond wrote:
> >The JSON format you mentioned looks reasonable. However, this brings us
> >back to the infrastructure question: we'll need to host a corresponding
> >CGI (or whatever it is that implements the query service) somewhere.
> >Where?
> 
> Even though I called it an "API", you probably could implement this as
> static JSON files without any CGI or dynamic calculations.

Right, very good point.

> Every time you publish new AMIs, write new static JSON files with the
> latest ids.

Anders, James, do you think this is easy to integrate in the AMI build
toolchain? If you can generate the static JSON file at each AMI
generation / upload, I can then investigate with the www.d.o team how to
publish it on www.d.o natively, if feasible there, or where else it
should go.

TIA,
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  zack@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Debian Project Leader . . . . . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »
[signature.asc (application/pgp-signature, inline)]

Information stored :
Bug#694035; Package cloud.debian.org. (Sat, 24 Nov 2012 14:15:28 GMT) Full text and rfc822 format available.

Acknowledgement sent to Anders Ingemann <anders@ingemann.de>:
Extra info received and filed, but not forwarded. (Sat, 24 Nov 2012 14:15:28 GMT) Full text and rfc822 format available.

Message #15 received at 694035-quiet@bugs.debian.org (full text, mbox):

From: Anders Ingemann <anders@ingemann.de>
To: debian-cloud@lists.debian.org
Cc: 694035-quiet@bugs.debian.org
Subject: Re: JSON based API to query available AMIs
Date: Sat, 24 Nov 2012 15:14:24 +0100
> toolchain? If you can generate the static JSON file at each AMI
> generation / upload, I can then investigate with the www.d.o team how to
> publish it on www.d.o natively, if feasible there, or where else it
> should go.
Shouldn't be a problem, but I am not sure how the procedure of
replicating the AMIs across regions work.
It might be out of the scope of the bootstrapper to generate the full
JSON object.
If we could automate the replication across regions and the publishing
process, then yes; we could generate the whole document.


Anders


On 24 November 2012 11:35, Stefano Zacchiroli <zack@debian.org> wrote:
> [ adding the bug log to Cc, to keep track of relevant information ]
>
> On Thu, Nov 22, 2012 at 10:52:46PM -0800, Eric Hammond wrote:
>> >The JSON format you mentioned looks reasonable. However, this brings us
>> >back to the infrastructure question: we'll need to host a corresponding
>> >CGI (or whatever it is that implements the query service) somewhere.
>> >Where?
>>
>> Even though I called it an "API", you probably could implement this as
>> static JSON files without any CGI or dynamic calculations.
>
> Right, very good point.
>
>> Every time you publish new AMIs, write new static JSON files with the
>> latest ids.
>
> Anders, James, do you think this is easy to integrate in the AMI build
> toolchain? If you can generate the static JSON file at each AMI
> generation / upload, I can then investigate with the www.d.o team how to
> publish it on www.d.o natively, if feasible there, or where else it
> should go.
>
> TIA,
> Cheers.
> --
> Stefano Zacchiroli  . . . . . . .  zack@upsilon.cc . . . . o . . . o . o
> Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
> Debian Project Leader . . . . . . @zack on identi.ca . . o o o . . . o .
> « the first rule of tautology club is the first rule of tautology club »



Information stored :
Bug#694035; Package cloud.debian.org. (Sun, 25 Nov 2012 13:57:12 GMT) Full text and rfc822 format available.

Acknowledgement sent to James Bromberger <james@rcpt.to>:
Extra info received and filed, but not forwarded. (Sun, 25 Nov 2012 13:57:12 GMT) Full text and rfc822 format available.

Message #20 received at 694035-quiet@bugs.debian.org (full text, mbox):

From: James Bromberger <james@rcpt.to>
To: debian-cloud@lists.debian.org
Cc: 694035-quiet@bugs.debian.org
Subject: Re: JSON based API to query available AMIs
Date: Sun, 25 Nov 2012 21:31:28 +0800
[Message part 1 (text/plain, inline)]
On 24/11/2012 10:14 PM, Anders Ingemann wrote:
>> toolchain? If you can generate the static JSON file at each AMI
>> generation / upload, I can then investigate with the www.d.o team how to
>> publish it on www.d.o natively, if feasible there, or where else it
>> should go.
> Shouldn't be a problem, but I am not sure how the procedure of
> replicating the AMIs across regions work.

There;s two approaches. As each AMI has a unique AMI ID and is a fresh
snapshot and registration in each region, you can either copy a master
AMI's snapshot, or re-run the bootstrap script in each Region. As
everything is an API call in AWS, you can do both programatically; my
preference has been to re-bootstrap; doing this was far faster than
copying one snapshot around.

Indeed, the first step in this was using the quotes script in
http://wiki.debian.org/Cloud/AmazonEC2Image/Squeeze that installs git,
clones the bootstrap script, creates a 32 and 64 bit image, and then
shuts down! This could be make into an API call from the command line,
and then just point the script at each Region - and a shell script to
iterate through each region (in parallel?).

Any AMI list could be done as a CloudFront template - see the example on
http://wiki.debian.org/Cloud/AmazonEC2Image/Squeeze. This example can be
put into an AWS Bucket for public distribution. We could also chose to
publish the "mappings" section similarly for inclusion into other templates.

> It might be out of the scope of the bootstrapper to generate the full
> JSON object.
> If we could automate the replication across regions and the publishing
> process, then yes; we could generate the whole document.

Agreed; makes sense to wrap that all up in one "deploy new release to
all regions" script.



-- 
/Mobile:/ +61 422 166 708, /Email:/ james_AT_rcpt.to
PLUG President 2012: http://www.plug.org.au <http//www.plug.org.au>
[Message part 2 (text/html, inline)]

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Cloud Team <debian-cloud@lists.debian.org>:
Bug#694035; Package cloud.debian.org. (Sun, 06 Jan 2013 02:03:06 GMT) Full text and rfc822 format available.

Acknowledgement sent to Charles Plessy <plessy@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Cloud Team <debian-cloud@lists.debian.org>. (Sun, 06 Jan 2013 02:03:06 GMT) Full text and rfc822 format available.

Message #25 received at 694035@bugs.debian.org (full text, mbox):

From: Charles Plessy <plessy@debian.org>
To: 694035@bugs.debian.org
Subject: Re: Bug#694035: JSON based API to query available AMIs
Date: Sun, 6 Jan 2013 10:59:15 +0900
Le Sun, Nov 25, 2012 at 09:31:28PM +0800, James Bromberger a écrit :
> 
> Any AMI list could be done as a CloudFront template - see the example on
> http://wiki.debian.org/Cloud/AmazonEC2Image/Squeeze. This example can be
> put into an AWS Bucket for public distribution. We could also chose to
> publish the "mappings" section similarly for inclusion into other templates.

Hi James and everybody,

in order to experience myself with parsing remote JSON data, I have copied the
mapping of image identifiers of Cloud/AmazonEC2Image/Squeeze to
Cloud/AmazonEC2Image/Squeeze/JSON.

Here is what I added to the Cloud/AmazonEC2Image/Squeeze page.

    It is planned to provide a machine-readable version of the above list of
    images. For the sake of the brainstorm, a JSON version is temporarly placed at
    Cloud/AmazonEC2Image/Squeeze/JSON. However, the structure is very likely to
    change, see #694035 for details. Here is a naive example on how to query the
    list. euca-describe-images $(curl --silent
    http://wiki.debian.org/Cloud/AmazonEC2Image/Squeeze/JSON?action=raw | jsonpipe
    | awk '/Debian606.ap-northeast-1.64/ {print $2}' | sed 's/"//g')

Have a nice week-end,

-- 
Charles Plessy
Tsurumi, Kanagawa, Japan



Information forwarded to debian-bugs-dist@lists.debian.org, Debian Cloud Team <debian-cloud@lists.debian.org>:
Bug#694035; Package cloud.debian.org. (Mon, 07 Jan 2013 04:39:03 GMT) Full text and rfc822 format available.

Acknowledgement sent to "Bromberger, James" <jameseb@amazon.com>:
Extra info received and forwarded to list. Copy sent to Debian Cloud Team <debian-cloud@lists.debian.org>. (Mon, 07 Jan 2013 04:39:03 GMT) Full text and rfc822 format available.

Message #30 received at 694035@bugs.debian.org (full text, mbox):

From: "Bromberger, James" <jameseb@amazon.com>
To: Charles Plessy <plessy@debian.org>, "694035@bugs.debian.org" <694035@bugs.debian.org>
Subject: RE: Bug#694035: JSON based API to query available AMIs
Date: Mon, 7 Jan 2013 04:37:42 +0000
Nice.

Only question - with marketplace not available in ap-southeast-2 (Sydney), do we want ot put in the AMIs for the 32 and 64 bit AMIs that sit inside our Debian AWS  Account?

  James

James Bromberger | Solution Architect | Amazon Web Services
E: jameseb@amazon.com   P: +61 422 166 708



-----Original Message-----
From: Charles Plessy [mailto:plessy@debian.org] 
Sent: Sunday, 6 January 2013 9:59 AM
To: 694035@bugs.debian.org
Subject: Bug#694035: JSON based API to query available AMIs

Le Sun, Nov 25, 2012 at 09:31:28PM +0800, James Bromberger a écrit :
> 
> Any AMI list could be done as a CloudFront template - see the example 
> on http://wiki.debian.org/Cloud/AmazonEC2Image/Squeeze. This example 
> can be put into an AWS Bucket for public distribution. We could also 
> chose to publish the "mappings" section similarly for inclusion into other templates.

Hi James and everybody,

in order to experience myself with parsing remote JSON data, I have copied the mapping of image identifiers of Cloud/AmazonEC2Image/Squeeze to Cloud/AmazonEC2Image/Squeeze/JSON.

Here is what I added to the Cloud/AmazonEC2Image/Squeeze page.

    It is planned to provide a machine-readable version of the above list of
    images. For the sake of the brainstorm, a JSON version is temporarly placed at
    Cloud/AmazonEC2Image/Squeeze/JSON. However, the structure is very likely to
    change, see #694035 for details. Here is a naive example on how to query the
    list. euca-describe-images $(curl --silent
    http://wiki.debian.org/Cloud/AmazonEC2Image/Squeeze/JSON?action=raw | jsonpipe
    | awk '/Debian606.ap-northeast-1.64/ {print $2}' | sed 's/"//g')

Have a nice week-end,

--
Charles Plessy
Tsurumi, Kanagawa, Japan


-- 
To UNSUBSCRIBE, email to debian-cloud-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20130106015915.GA21877@plessy.org




Send a report that this bug log contains spam.


Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Mon Apr 21 13:16:07 2014; Machine Name: buxtehude.debian.org

Debian Bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.