Saturday, August 2, 2014

The flexible form builder [Part 1 viewer]

Any html form is a recursive structure of fields. A field type can be one of many primitive types like text, textarea, select, radio etc. or composite type that combines primitive types or other composite types or array of primitive or composite types. Form values are saved as a single row which is a stored composite object like SQL table row or a JSON object. Field values can be a reference to another stored object.

We want a form designer that has full support for all types of fields. So, we need support for primitive, composite and array types and any combination of those types.

As first step I have designed a form viewer that can render the form from a form definition provided as JSON object. To keep the blog post simple I use one primitive type named text, composite type, array type and to define a form a separate form type. Form , composite and array types has child fields array named fields.

Here is an example of possible form definition:

form =
    {'name':'form', 'type':'composite', 
        'fields': [    
            {'name': 'name','type': 'text'},
            {'name':'address', 'type':'composite',
               'fields':[
                   {'name': 'street','type': 'text'},
                   {'name': 'city','type': 'text'},
                   {'name': 'zip','type': 'text'},
                   {'name': 'state','type': 'text'},
                   {'name':'members', 'type':'array',
                       'fields':[
                            {'name': 'name','type': 'text'}
                        ]
                   }
               ]          
            },
            {'name':'assets', 'type':'array',
                'fields':[
                    {'name': 'type','type': 'text'},
                    {'name': 'price','type': 'text'},
                    {'name': 'brand','type': 'composite', 
                        'fields':[
                            {'name': 'name','type': 'text'},
                            {'name':'locations', 'type':'array',
                                'fields':[
                                    {'name': 'city','type': 'text'}
                                 ]
                            }
                        ]
                    }       
                ]          
            }       
        ]
    };  

This is a form that has primitive types, composite type, array of primitive type and array of composite type. Other than reference type which is just a primitive type with loaded data this covers what we need to create almost any type of form.

Here is the form viewer preview:
See the Pen atCcI by Maruf Maniruzzaman (@kuasha) on CodePen.
I have created an angular directive named field.
cosmosApp.directive('field', function($compile) {
    return {
      restrict: 'E',
      scope: {
        item:'='        
      },
      
      link: function(scope, element, attributes) {
        scope.item_data = [];
        scope.add_item = function(position){
          var newItem = {};
          angular.forEach(scope.item.fields, function(value, index){
            newItem[value.name] = "test";
          });
          scope.item_data.splice(position+1, 0, newItem);
                          
        };
        scope.removeItem = function(index){
          scope.item_data.splice(index, 1);
        };
        var template = '<span>{{item.name}}<input type="text" /></span> &nbsp;'; 
        
        if(scope.item.type === "composite"){
          template = '<div>{{item.name}}</div><ul><li ng-repeat="ph in item.fields">
                            <field item="ph"></field></li></ul>';
        }
        else if(scope.item.type === "array"){
          if(scope.item_data.length<1){
            scope.item_data = [];
            scope.add_item();
          }
          template = '<div>{{item.name}}<button ng-click="add_item(-1)">+</button></div>
                         <ul><li ng-repeat="d in item_data">
                           <field item="ph" ng-repeat="ph in item.fields"></field>
                           <button ng-click="removeItem($index)">-</button>
                           <button ng-click="add_item($index)">+</button></li>
                         </ul>';
        }
        
        var newElement = angular.element(template);
        $compile(newElement)(scope);
        element.replaceWith(newElement);        
      }
    };
  });
To use this directive we create a form in our controller and create a dom node like this:
    

Friday, July 11, 2014

Continuous integration using Travis CI

It has been tough keeping the frameworks functionality working after the basic prototype. The prototype is great - it has most basic features. But adding new feature or fixing bugs was breaking functionality here and there. So, instead of coding new functionality I have added about 21 tests. Most of them are functional tests at this moment. And while writing them design and functionality bugs were revealed and fixed. I strongly feel we need more tests before adding any more functionality, but for now the tests are covering most of the code path and can fix remaining bugs and start writing unit tests.

I have added the project to Travis CI which is great free tool to build and integration test. The following image shows current status of build/ test run:

Build Status

It runs every time a new push is made to the Github repository. This image is actually dynamic and shows current status of Cosmos Framework. If you can not see it look here.

Setting up TravisCI is easy. First login to https://travis-ci.org/ using your Github account and Travis CI will show the list of public repositories. Now enable any repository Travis should track. Then we need to add a file named .travis.yml at the root of the repository. This file contains basic settings like language and script to run. For cosmos we need MongoDB which can also be added usnder required services. Here is the first version of the configuration file being currently used to run build and tests:


language: python
python:
  - "2.7"
# command to install dependencies
install:
  - pip install tornado
  - pip install motor
  - pip install mongolog
  - pip install mock
  - pip install requests

services:
  - mongodb

before_script:
#This mongodb command should not have a newline in real file
- mongo 127.0.0.1/cosmos --eval 'db.cosmos.users.insert(
     {
         "username":"admin",
         "roles": ["43425097-e630-41ea-88eb-17b339339706"],
         "email":"admin@cosmosframework.com",
         "password":"hmac:1ddb4e6b0b810f59018babbb5dc0eed4"
     });'

# command to run tests
script: python setup.py test

We are installing the required python packages and adding mongodb as required service. Before running script we want to add our admin user to mongodb which is done in before_script section.

When we add this file, commit and push the change to Github, Travis gets notification using webhook about the change and start deploying a system and run tests. It shows the status using an image (as shown above) that changes based on the build/test result.

This is a very nice free service which you may use for your opensource project and for each change it will run your tests and report if anything is broken.

Wednesday, July 2, 2014

How to create angular app and host on heroku using cosmos frameowk in 5 simple steps

Angular seed project is configured to use npm to do basic tasks like download required packages. You should have node installed on your machine. You won't need it after you download the required package. Cosmos will host the application using tornado in both development and production environment. In production use a reverse proxy like nginx to serve the static files. Now follow the steps to create and run the angular-seed application.

Create angular project using cosmosadmin

If you do not have mongodb or virtualenv installed install those first. For ubuntu its like:

sudo apt-get install mongodb
sudo apt-get install python-virtualenv

First create a virtualenv and activate it:

virtualenv --no-site-packages testenv
source testenv/bin/activate
Now install cosmos inside it:
pip install cosmos

Run new-project command with angular from console from any directory to create new project:

cosmosadmin new-project angular

This will create basic cosmos application and download angular-seed project from github. Here is sample output.

new-project
-----------Cloning angular seed project--------------

Cloning into 'angular-seed'...
remote: Counting objects: 2475, done.
remote: Compressing objects: 100% (1237/1237), done.
remote: Total 2475 (delta 1071), reused 2475 (delta 1071)
Receiving objects: 100% (2475/2475), 13.80 MiB | 2.19 MiB/s, done.
Resolving deltas: 100% (1071/1071), done.
Checking connectivity... done.
----------- You should run "npm install" from angular-seed directory now -------------

You should now open settings.py and local_settings.py files to adjust database and other settings.

Now change directory to angular-seed and run npm install:

cd angular-seed
npm install

This will download all required packages for angular-seed application.

Now go back to project directory and run:

cd ..
python cosmosmain.py

You may now browse to http://localhost:8080/ to view your newly created application.

Host the app on heroku

If you want to host your application on heroku you need Procfile and requirements.txt files. You may create these files using following command:

python cosmosmain.py add-herokusettings

Now run the application using foreman (you should have heroku toolbelt installed on the machine):

foreman start

You may now browse to http://localhost:5000/ to view your application. Now commit and push the application to heroku for deployment.

Tuesday, July 1, 2014

How to install cosmos framework

The framework is uploaded to python package index to make it simple to install as library. It is recommended that you use virtualenv, but it is not required.

First create a virtualenv and activate it:

virtualenv --no-site-packages testenv
source testenv/bin/activate
Now install cosmos inside it:
pip install cosmos
A console script named cosmosadmin has now been created for administration tasks. Run new-project command from console from any directory to create new project:
cosmosadmin new-project

The app folder is the root of the webserver in the create project by default. Index page is mapped to serve templates/index.html with current_user passed to it. The /service/* endpoint is mapped to the service handler. You may look into endpoints.py for other endpoints.

After creating the project change the values in the settings.py file (or create a new file named local_settings.py nad add it to your .gitignore file- so this settings does not end up in the source repository) for the database and create admin user:

python cosmosmain.py new-admin
Now start the project:
python cosmosmain.py start-service
And browse to http://localhost:8080/ to see the start page.

Sunday, June 29, 2014

Create a user, a role and assign the role to the user

Out of the box Cosmos framework comes with a set of RESTful APIs to manipulate objects. The GET, POST, PUT and DELETE http methods are used to manipulate objects. Those are treated as read, insert, update and delete respectively. The data, where applicable, is sent as JSON objects in the request body of the request. There are some predefined role and role group objects out of the box. Other than those the default APIs and sample front end project can be used to create/read/edit/delete any object in the system. Developers should be able to create javascript application to manipulate any object on the server without writing any server side code.

Create a user

The object name for default user store is cosmos.users. To create a user an administrator must have access to this object like any other object. User, like any other object, can be added using a POST request to the endpoint /service/cosmos.users/. Two fields must be present for any user to login: username and password. But a user will typically also have an email address and a list of roles assigned. Here is a sample object:

{
    "username":"scientist",
    "password":"secret",
    "email":"scientist@copotron.com",
    "roles":[]
}

When this object is posted the password field will be replaced by the cosmos.users preprocessor and convert it to hmac value of the password and add "hmac:" at the start before saving to database. If the password is already hashed and the value starts with "hmac:" it will not be converted again. But without the "hmac:" the value will be converted for both POST and PUT methods. When we get the user back using GET method it will be something like:

{
  "username": "admin",
  "email": "admin@cosmosframework.com",
  "roles": [
    "43425097-e630-41ea-88eb-17b339339706"
  ],
  "password": "hmac:1ddb4e6b0b810f59018babbb5dc0eed4",
  "createtime": "2014-06-28 19:07:16.127194",
  "_id": "53af74d48c66ab119e60c2d7",
}

The _id field is the string representation of the mongodb ObjectId identifier for this item in the object collection and we can use this value while using PUT method.

Create a role

The object name for default role store is cosmos.rbac.object.role. An administrator may POST following object to create a role:

{
    "name": "TestRole", 
    "role_items": [
        {
            "access": [
                "INSERT", 
                "READ", 
                "WRITE", 
                "DELETE"
            ], 
            "object_name": "testservice", 
            "property_name": "*", 
            "type": "object.RoleItem"
        }
    ], 
    "sid": "984fdf54-0096-4a95-adff-4c791ad240ed", 
    "type": "object.Role"
}

The role_items is an array of access role. role_items.access is an array of access type for this role item. The role_items.object_name creates access for the named object and role_items.property_name creates access for the property of the object. These two can be think of as table/field combination in database terms. Either of them can be set to * which will match all objects or property. You should be careful while assigning this to role_items.object_name. By default the builtin SYSTEM user has set both of the fields to wild card access. The sample role above gives access to all fields of testservice object to the user who is assigned this role.

Note here that you may assign a sid while creating the role object, but this is optional. The role object has a preprocessor which will assign this automatically if it is left empty.

Assign a role to a user

An administrator can create a user roles[] assigned using POST method or later can use PUT method to update the role. Here in our example since we have created the user first without any role we will use PUT method to update the roles of the user. The endpoint has for the user object will be /service/cosmos.users/53af74d48c66ab119e60c2d7/. The value 53af74d48c66ab119e60c2d7 is the same _id of the users mongodb ObjectId. And the body of the request will be:

{
  "roles": [
    "43425097-e630-41ea-88eb-17b339339706"
  ]
}

We use the sid value of the role (or rolegroup). When the roles for the user is updated the user can now perform operations given by the access on the object and property.

We will discuss about more advanced role access where you can allow permission using field name instead of widcard access and more advanced access like address.city in the next article.

Tuesday, June 17, 2014

Cosmos Web Framework Introduction

A python web framework with event based auto refresh and zero code service framework that is protected by Role Based Access Control for each object (table) and property (field). Even when a field has nested field (like address.city) those can also be protected individually or collectively for insert/update/delete operation. This video shows the event based update- updates only on change. No periodical polling.


Here is a demo that shows the data synchronization between multiple user sessions (bootstrap example project does not have the functionality):
In near future we will be publishing other features and commit the framework to github.