JeeWiz

3 Posts
Matthew

I was at the London CloudCamp last night, with a large crowd, mostly techies. Gojko Adzic has posted a comprehensive reviewalready!

 

I ran an open space (break-out) session on persistence in cloud applications. As Gojko points out, we didn't have time to get into as much detail as I'd hoped - in fact, we didn't even get through my list of questions! However, the fact that the largest number of people were interested in persistence in clouds was the most interesting feedback - at least I'm not alone in puzzling over this area!

 

As we didn't even finish the introduction and list the questions(!) during the session, here's a brief summary of the background:

  • My company's interest is in offering consultancy services in this area, so I'd like to know what we're talking about, surprisingly.  We also produce model-driven generators for application development, and we have a version for the GigaSpaces platform called GigaSystemBuilder: this just does the nuts and bolts right now, no persistence like the others.
  • GigaSpaces is somewhat different from other grid/cloud solutions in that it focusses on grid components that integrate both data and processing - see Nati Shalom's explanation.  It also provides partitioned clustering that is handled by the container, so the power of a cluster can be beefed up at run-time if the system gets hot.
  • So the stage seems to be set to handle really fast data-oriented applications.  But my feeling - shared with others, if discussions after the session are anything to go by - is that there is still a gap between an enterprise application developer's view of how to handle data and the grid vendors'.  An impedance mismatch, as it were.
  • Seasoned campaigners of large-scale distributed computing like Eric Brewer and Pat Helland tell us that programming for large-scale, distributed applications is a different kettle of fish to your dad's J2EE.
  • So the master question was, what will the dominant architecture be in 5 years time for cloud/grid applications.  And this becomes really interesting if the architecture shows how to do data persistence at grid-like speeds, but with traditional database systems' reliability.

There does seem to be a "missing link" (as one participant put it) here - a layer between the classical 'entity' layer and a physical persistence service, to handle the  details of distributed multi-layer transactions for data operations, plus miscellaneous twiddly bits (e.g. allocating unique sequence numbers across a partitioned architecture).  Hopefully we can make this area easy enough for take-up by large numbers of application developers, and simple enough that they can produce a fast scalable architecture without too many tears (or tiers).

 

If you have an interest in this area, let me know and I'll share more of the detail behind this.  Hopefully I'll be posting a thought-out paper in a couple of weeks.

0 Comments 0 References Permalink Tags: london, cloudcamp, persistence, j2ee, gigaspaces, gigasystembuilder
ian

This blog descibes my experiences in getting a GigaSpaces application deployed to the EC2 cloud.

 

My starting point was to use the GigaSpaces documentation at http://www.gigaspaces.com/wiki/display/XAP66/EC2+Framework+Developers+Guide. I also needed the Amazon EC2 Getting Started Guide - http://docs.amazonwebservices.com/AmazonEC2/gsg/2007-01-19/?ref=get-started and the Amazon EC2 Developer Guide - http://docs.amazonwebservices.com/AmazonEC2/dg/2006-10-01/.

 

On the whole the deployment went pretty smoothly but I did have some head-scratching in a few areas.  What I'll do now is highlight those areas, then give the full step-by-step instructions.

 

Issue 1 - Adding ports to security groups

One of prerequisites in The Framework Developers Guides is to add various ports to the security groups.

 

             Add ports 22, 80, 442 to your EC2 security groups. Click here to do this.
             Port 22 - used for ssh communication
             Port 80 - used for deploying the GigaSpaces Web Management Center and the Ganglia monitoring tool
             Port 443 - used for secured http.

 

First of all, there is a typo here - "442" is wrong - "443" is correct, being the default port for http is 443.

 

The second part of this issue is that it was unclear what security groups I should add these three ports to. As far as I could see there were no security groups in my set up.

I created a new group to complete this step, but it turns out the easist was to do this is to use the default group, conveniently named 'default'. So the commands in this step are

 

             <ec2-api-tools-install-dir>\bin\ec2-authorize default -p 22

             <ec2-api-tools-install-dir>\bin\ec2-authorize default -p 80

             <ec2-api-tools-install-dir>\bin\ec2-authorize default -p 443

 

Setting the ports to the wrong or invalid security group cause problems further down the line.

In my case the symptom was that the UI web server never started properly and so I was unable to start the GigaSpaces Management Centre:

     the gs-launch-cloud.cmd reported 'UI server timed out'.

 

             - Waiting for ui web server to start.
             ..
             - Time: 5sec

 

             - Time: 31sec
             - UI server timed out

 

This is a bit confusing because the UI AMI does start and has a public DNS ... but then this "hang around and report nothing" is typical of security errors!

 

Issue 2 - <start-ui>true

The application I tried to deploy was the 'data-example' provided in the gigaspaces-cloud-tools-v.2.0.0-beta zip file.

 

The 'data-example' config file 'data-example.xml' has the value of the <start-ui> to be 'true'. The Framework Developers Guide lists four options for <start-ui>, which don't include 'true' (but does include false).

Having solved issue 1, I changed the <start-ui> value to be 'admin-application' - this works well.

 

Issue 3 - Starting the GigaSpaces Management Tool.

 

The Framework Developers Guide also has a section which explains why the Management Tool sometimes doesn't start automatically. The solution is to paste the URL into a browser.

 

In my investigations to solve Issue 1 I did in fact download the NX Client. The data-example.xml incorporates a default-settings config file.

 

          C:\gigaspaces-cloud-tools-v2.0.0-beta\default-settings\cloud-config.xml

 

This config file has included in it a <nx-command> which was originally set to "D:\Tools\NXClient\nxclient.exe". Obviously you need to change this to point to your installation of the NX Client.

 

Once I had downloaded NX Client and corrected the location, I no longer had any problems with the Management Tool starting automatically.

     Bottom line: if you do have problems w GS Mgmt tools starting, set up NX Client.

 

Step-by-step Instructions

This section simply shows the steps I used. There is no new documentation here - most of the notes have been taken directly from the GigaSpaces EC2 Framework Developers Guide, the EC2 Getting Started Guide and EC2 Developers Guide - but it does give a coordinated view of the install across the three sets of documentation.

 

1. Create an Amazon Web Services (AWS) account: http://www.amazon.com/gp/aws/registration/registration-form.html

          You will become the proud owner of an "AWS account ID".  Remember it.

 

2. Sign up to EC2 and S3 from within your amazon account. You will need to provide credit card details.

 

    For EC2 you need to set up some access keys and x509 certificate. **Note these numbers down carefully.**

 

         2.1  Access Key Info
         ===============

          This can be obtained at:
             https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action =access-key

 

          Your Access Key ID = 1NBPS61PV6MF2D87AHGR2

          Secret Access Key  = bM9k/Ei6tGhsfuYaMrKK/jSpLno9n+376po25enc8

 

       2.2  X509 Certificate
        ===============

          It really helped to read the following which explains how to get and store the x509 certificate

             http://docs.amazonwebservices.com/AmazonEC2/gsg/2007-01-19/?ref=get-started

 

          Click on 'Setting up an Account' navigation bar on the LHS of the above doc to explain how to create and store the x509 certificate


          The x509 certificate can be obtained from
             https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action =access-key - this is the same link in the "Access Key Info" above.

 

          By following these instructions you will end up with 2 files - pk???.pem and a cert???.pem.

          Create a directory '.ec2' in your home directory. Store the 2 files in .ec2 on your home directory, e.g. on Windows and using sample filenames - your filenames will differ.
              C:\Documents and Settings\ian\.ec2\pk-22SAD7AOV4OQPTJPAYRH2XAZTXBULZ2E.pem
              C:\Documents and Settings\ian\.ec2\cert-22SAD7AOV4OQPTJPAYRH2XAZTXBULZ2E.pem

          On Unix, use '~' instead of my home directory (~/.ec2/...).

 

        2.3  EC2 Account - AWS Account ID
        ===========================
          When you signed up to AWS in step 1, you got your AWS account ID.  Your EC2 user ID is the AWS account ID with the hyphens removed

              AWS Account ID = 3211-7833-2797
              EC2 Account ID = 321178332797


3. Amazon EC2 Tools Setup.

     In this section you set up the EC2 tools and tell them where to find the x509 certificate and the EC2 private key.

 

    Download the tools from here

         http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351&categor yID=88

 

     and extract.  We'll call the directory for the Amazon EC2 tools '<amazon-ec2-api-tools-install-dir>'.

 

     There's a bit of confusion here:

          - in some of the documentation it says to store these in the .ec2 directory in your home directory (item 2 above)

          - in this section of the documentation it says to store them in the EC2 tool directory.

     I did both - it worked for me!!

 

    The EC2 command tools need the x509 certificate (see previous step). The details can be found here
         http://docs.amazonwebservices.com/AmazonEC2/gsg/2007-01-19/?ref=get-started.

     Then click 'Setting up the Tools'

 

     The settings I used are below. I put them into window environment variables.
         set EC2_HOME=C:\ec2-api-tools-1.3-24159
         set PATH=%path%;%EC2_HOME%\bin
         set EC2_PRIVATE_KEY=C:\ec2-api-tools-1.3-24159\pk-22SAD7AOV4OQPTJPAYRH2XAZTXUBLZ2E. pem -- use your filename here
         set EC2_CERT=C:\ec2-api-tools-1.3-24159\cert-22SAD7AOV4OQPTJPAYRH2XAZTXUBLZ2E.pem -- use your filename here

 

4.  Configure the GigaSpaces tools.

 

     4.1 Sign up to GigaSpaces and download tools

     ==================================

     Sign up to GS at

         http://www.gigaspaces.com/ec2signup

 

          This will result in an email being sent to you with a GigaSpaces cloud license code in it.

    
     Download the following zip file and extract
         http://www.gigaspaces.com/tempfiles/ec2/gigaspaces-cloud-tools-v2.0.0-beta.zip
   
    The following description assumes you unzip this into
         C:\gigaspaces-cloud-tools-v2.0.0-beta

 

    4.2 Access Key ID
    ==============
     Create a file called 'key.txt' in C:\gigaspaces-cloud-tools-v2.0.0-beta\default-settings\keys.

     This is to hold your public key for encryption.

     The value must be equal to your Access Key ID (see step 2.1).
          Make sure there is no whitespace (spaces or newlines) before or after the Key string - so just '1NBPS61PV6MF2D87AHGR2'.
   
    4.3 Secret Access Key
    =================
     Do the same thing as in 4.2, but for the Secret Access Key (see step 2.2) using the file name secret-key.txt ... so create \gigaspaces-cloud-tools-v2.0.0-beta\default-settings\keys\secret-key.txt.
     Same deal about no extraneous whitespace.
   
    4.4  Cloud License Key
    ==================

     When you first download it, the file C:\gigaspaces-cloud-tools-v2.0.0-beta\default-settings\gslicense.xml looks like this

          <com>
          <j_spaces>
           <kernel>
              <!-- can also be set via system property using -Dcom.gs.licensekey -->
              <licensekey>Nov 20, 2008~GigaSpaces Eval Cloud@rgPn7FPQaN9MNSdPhW1N#EE^6.0XAP%UNBOUND</licensekey>
             </kernel>
          </j_spaces>
          </com>

     Replace the <licensekey> value with the license key cloud license secret-key from the email GigaSpaces sent you.
     
    4.5  Adding ports to EC2 security groups
    ===============================
    Add ports 22, 80, 442 to your EC2 security groups. This is done with the Amazon EC2 tools (from step 3)
   
             <amazon-ec2-api-tools-install-dir>\bin\ec2-authorize default -p 22

             <amazon-ec2-api-tools-install-dir>\bin\ec2-authorize default -p 80

             <amazon-ec2-api-tools-install-dir>\bin\ec2-authorize default -p 443

 

    Port 22 - used for https communication
    Port 80 - used for deploying the GigaSpaces Web Management Center and the Ganglia monitoring tool
    Port 443 - used for secured http.
       
5. Starting the cloud.

 

     I used the data-example which can be found in

          C:\gigaspaces-cloud-tools-v2.0.0-beta\examples\data-example

 

     To deploy the data-example into the cloud open a cmd box, change directory to

          C:\gigaspaces-cloud-tools-v2.0.0-beta\bin\

 

     and run the command

          gs-launch-cloud.cmd ..\examples\data-example\data-example.xml

 

     The log output is shown below.

 

OUTPUT FROM THE LOG
===================
-------------------------------------------------------------------------------- -----------------------------------------------------------
- Gigaspaces Cloud Tools v2.0.0 Beta
- loading Config File : ..\examples\data-example\data-example.xml
- loading Config File : C:/gigaspaces-cloud-tools-v2.0.0-beta/bin/../examples/data-example/../../defaul t-settings/cloud-config.xml
- Command: createCloud
- Cloud name: data-example-ian
- Number of args: 0 : []
- AMI: ami-6adc3803 Type: small
- Cloud Keys Validated.
- Gigaspaces License Validated.
- Deleted cloud data from repository
- Running Instances
- Number of GSM machines    : 1
- Number of GSM per machine : 1
- Number of GSC machines    : 3
- Number of GSC per machine : 2
- Number of GSC to wait before we deploy : 6
- Using Load balancer       : null
- Using Database            : false
- Putting Jars in Repository
.
- Time: 2sec
- Waiting for Cloud GSM to start
................................................................................ ..
- Time: 204sec
- Could not find existing UI machine - starting up a new one
- Using GSM running on:domU-12-31-39-00-C6-45.compute-1.internal
................................................................................
- Time: 198sec
- Ui machine public dns = ec2-75-101-193-110.compute-1.amazonaws.com
- Waiting for Cloud GSC to start
...
- Time: 5sec
- Cloud GSM started successfully on the following machine IP: domU-12-31-39-00-C6-45.compute-1.internal
-       GSM Public Dns: ec2-75-101-183-185.compute-1.amazonaws.com
- Cloud GSCs started successfully on the following machines IP:
        ec2-75-101-230-41.compute-1.amazonaws.com
        ec2-75-101-240-115.compute-1.amazonaws.com
        ec2-67-202-31-176.compute-1.amazonaws.com
- Waiting for ui web server to start.
..
- Time: 5sec

 

- Time: 31sec
- UI server is ready
- Deploying Processing Units.
..............................
- Time:74sec
-
-  instance id , type ,    public dns            &nbs p;                 ,    private dns            &nbs p;            , status   , ami id
  i-1901af70 ,  gsm ,  ec2-75-101-183-185.compute-1.amazonaws.com ,   domU-12-31-39-00-C6-45.compute-1.internal , running , ami-6adc3803
  i-1b01af72 ,  gsc ,   ec2-75-101-230-41.compute-1.amazonaws.com ,   domU-12-31-39-00-C4-B2.compute-1.internal , running , ami-6adc3803
  i-1a01af73 ,  gsc ,  ec2-75-101-240-115.compute-1.amazonaws.com ,   domU-12-31-39-00-A0-D3.compute-1.internal , running , ami-6adc3803
  i-1d01af74 ,  gsc ,   ec2-67-202-31-176.compute-1.amazonaws.com ,   domU-12-31-39-00-B1-92.compute-1.internal , running , ami-6adc3803
  i-901eb0f9 ,   ui ,  ec2-75-101-193-110.compute-1.amazonaws.com ,   domU-12-31-39-00-E0-86.compute-1.internal , running , ami-6adc3803

 

-------------------------------------------------------------------------------- -----------------------------------------------------------

 

6. Shutting down

 

     To shutdown the data-example into the cloud open a cmd box, change directory to

          C:\gigaspaces-cloud-tools-v2.0.0-beta\bin\

 

     and run the command

          gs-shutdown-cloud.cmd ..\examples\data-example\data-example.xml

 

          Your accountant will be very glad to hear you have shut down the cloud.

             Because if you don't, you will be charged by the instance-hour.   !!!

 

7. Monitoring

 

    As noted above, there are a number of options for monitoring tools.
   
    GigaSpaces Management tool

    ======================
          This is the one we used and it is the one you see on a GigaSpaces course.

          It should start automatically but see the issues at the start of this blog if it does not.
       
    Other tools
    ========   
    The monitoring tools are
    Firefox Organizer - to view deployment repository

             https://addons.mozilla.org/en-US/firefox/addons/policy/0/3247/20757
    Firefox ElasticFox - monitors active instance

             http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609
    Gangalia - monitoring system for clouds

             http://ganglia.info/

0 Comments 0 References Permalink
Jon

Shallow and Thoughtless

Posted by Jon Sep 26, 2008

I would have liked the first blog on this site to have been something deep and thoughtful on the subject of Code Generation and its growing place in IT provision, but that's not to be.  Instead I'm having to settle for a slightly less well thought out piece on whatever comes out as I type.  Please don't let me get blogger's block!  The boss said on Tuesday afternoon, can you get the community and support site up by Friday, which only just left me time for the mechanics and precious little left over for content.  Not enough time to buy hardware or software that's for sure, so it's been a case of make do, scouting out an underused PC, backing up and moving the data, reformatting the hard drives and installing the operating system, database, application server and community support software.  Then shlepping the whole thing upstairs to a different LAN, isolated from the development environment, and exposing it to the web.  I suppose one of the nice things about working in relatively small, developer-focused business is that you are allowed to do things like that, the disadvantage being you don't get much of a choice.

 

Thanks go to Jive at this point, who provide free licences for their Clearspace software for other open source sites like this.  Adding to that the MySQL community database server and Apache Tomcat, and we've a set of excellent products, all open source and free to use.  I didn't have quite enough time to shift everything over to Unix, or the whole set up would have been completely open.  Creating an open source release of JeeWiz doesn't feel quite so daring now as it did when it was suggested earlier this year.


It's an interesting exercise for someone of a certain age not just to move into the open source space, but to implement a community site to support it too.  It doesn't seem that many years ago that the social use of a company's e-mail system could get you fired, and now we are expected to take our water-cooler moments online.  We e-mail and blog and comment on forums, and if an Internet search doesn't provide the answer to all our questions, there's usually a helpful forum where others are blogging and commenting and will be only too willing to lend a hand.  Five years ago "they" were saying, be careful not to leave an online footprint, because when you go for a job they'll Google you and if they don't like what they see you won't get the job.  Now they are saying, make sure you have an online footprint, because if you're applying for a job and they can't Google you, you won't be seen as a team player.   We follow in the footsteps of journalists (how sad) as online participation is beginning to be seen as part and parcel of a career in IT.  Blogging is another one of those mental shifts we have to make as times change and we are called on to change with them: a strange melange of the personal and professional to suit the Facebook generation.  A good thing?  Quite possibly.  Time will tell, and not very much time at that.


I joined NT/e nearly five years ago, seduced by the quality and flexibility of the JeeWiz product, then out in the wild as version two point something.  Over the previous twenty years I'd seen application generation systems come and go and my opinion was always the same - more trouble than they're worth.  This was the first time I'd come across anything where I reckoned you'd get more out than you put in.  Since then we've worked on commercial transform stacks, creating full front-to-back systems in java, j2ee and c# .NET.  We've done web service applications from WSDL and web-service backbones for multi-national companies.  We've written transforms for Eclipse plugins and round tripped from IDEs to applications and back again.  We've tried our hand at 3GL to OO language transforms, automatic documentation generation, and websites.  We've built from DSL onto datagrids and generated a security system for a global company I'm not even allowed to mention. 


And now we've taken version five open source with jsf, Spring and Hibernate, and this has brought a whole new set of challenges, not least of which is how you support a product the core parts of which are free.  You are looking a part of our answer.  We'd love to create a community of JeeWiz users, both those who want to stretch the software in unexpected directions and those who just want to take the application generation stack and use it to churn out applications in half the time.  That way the next five years working with JeeWiz will be as exciting for us as the last.

 

 

Jon

0 Comments 0 References Permalink Tags: welcome, introduction, open, source

Actions

Recent Comments

No recent comments.

Popular Blog Posts