Developing Amazon Alexa Skills

The objective of this article is to explain what is Amazon Alexa, Alexa skills, how to build those skills. This article is intended for those who are interested in learning Alexa skill building and those who have some understanding of amazon lambda, NodeJs etc.

What is Amazon Alexa?
Let’s look at what Wikipedia has to say about Alexa. Alexa is an intelligent personal assistant developed by Amazon, first used in the Amazon Echo and the Amazon Echo Dot devices developed by Amazon Lab126. It is capable of voice interaction, music playback, making to-do lists, setting alarms, streaming podcasts, playing audiobooks, and providing weather, traffic, and other real-time information, such as news. Alexa can also control several smart devices using itself as a home automation system. Users can extend the Alexa capabilities by installing “skills”.

Why Voice?
Voice is the most natural form of user interaction, and recent advancements in speech recognition and natural language understanding technology are now enabling us to control our connected devices with speech. We believe that voice can simplify the modern lifestyle, providing us with a unifying interface across devices so we can be entertained, stay informed, and organize our lives by just asking.

What Is a Voice User Interface (VUI)?
A voice user interface (VUI) allows people to use voice input to control computers and devices. In short, it’s what enables voice experiences like the Star Trek computer and Alexa, Amazon’s voice service and the brain behind millions of devices including the Amazon Echo. Voice experiences are great when they offer a faster, easier, or more delightful way of doing things.

How to build custom Alexa skills?
This is a two-step process. First is to write a lambda or web service which will listen to Alexa events. We are going to write amazon lambda using NodeJs. And the second step is to configure skill events using Alexa Skills Kit. So how this works is, we create a new skill and provide the invocation name to it and some sample utterances. And these utterances are tied to an Amazon event which is handled by Amazon lambda. So when we wake Alexa with invocation name and ask something, based on utterance Alexa voice engine determines which amazon event to trigger, and output is provided by Amazon lambda function which handles that particular event.

 
Let me take you through the steps now:
1] Login to Amazon console and select lambda service. You will see the list of lambdas already created by you. You can see the Create Function button.

1

It is important to note that you have to select the region US East (N. Virginia) for Lambda functions used with the Alexa Skills Kit.

15

 
2] After clicking on Create Function button, you will be redirected to this page. Amazon supports lambda creation from blueprints or templates. For our skill, we are going to create it using blueprint.

2

 
3] Click on the blueprint and you will see the list of blueprints. Search for alexa-skill-kit-sdk-factskill and one blueprint will be shown below. Click on the skill and customize button on the right bottom.

3

 
4] Provide Basic information of lambda such as name, role etc and click on Create function button.

 
5] Now function has been created and configuration page will open. Now we have to add triggers to function. You can see the trigger list from the menu on left, click on Alexa Skill Kit to add it to the function. It is recommended that you add the Alexa skill trigger once you have set up an Alexa skill in developer console because you will have to provide the app id in the lambda skill trigger.
Now open the function code which is editable online and add following code in the handlers.

'DateIntent': function () {
  this.emit(':tell', 'You are asking for todays date.');
}

5

Proceed further and save the function.

 
6] Now, login to https://developer.amazon.com/ and click on the ALEXA tab inside. You will see something like this.

6

 
7] Now click on the Get Started button on Alexa Skills Kit section. And the Add a New Skill.
First, you have to provide the Skill information. Most important thing is to name the skill and provide the invocation name. Make sure not to use big invocation name and without any special characters. These guidelines are useful at the time of certification of the skill.
For example, if you provide invocation name as “date helper”. Then to invoke the skill, you will say something like Alexa, ask date helper to give me today’s date.

7

 
8] After saving the skill information, an ID will be allocated to the skill which you can provide in the lambda Skill Kit trigger which we discussed earlier.

 
9] Click next to go to Interaction Model. This is the section where you will be creating an Intent, utterances to invoke the Intent etc. Click on the “Launch Skill Builder” button which will take you to the dashboard.

 
10] On the left menu, you can see default intents which Amazon provides. Now you have to create ADD button to create custom intent.

10

 
11] Name the intent as DateIntent and click create intent button. Now it will ask for the sample utterances. You have to provide multiple utterances which could invoke/trigger this intent. The name of the intent should match with the handler you have written in lambda.

11

 

Intents can have slots configured. A slot is a keyword whose value is going to be dynamic. We can choose existing slot types or create new ones.

 

12] Save and build the model. Once the model is built, click on the configuration link to go to the previous page.

 

13] Configuration tab asks you for the service which is going to handle the intent events. Since are going to provide the AWS lambda ARN which is going to handle the intents. Click on the AWS Lambda radio button and provide lambda ARN which we have created earlier. Save and click Next.

 

14] Now you are onto the Test section which provides the voice simulator and service simulator. Provide sample utterance and click the Ask button. You will see the service request and response. You can copy this request which can be used in the lambda section to create tests.

14
Click Next and go to the Publishing Information section.

 
15] Publishing information asks you for the basic information about the skill. You will have to provide the category, testing instructions, description of the skill, sample utterances, and icons. Make sure to provide the correct description and sample utterances because those are important in case you try to submit the skill for certification.

 
16] Provide information in the privacy and compliance section and save. If you think your skill is good enough to go live, submit it for certification.

You can have a look at my GitHub repository to understand how the lambdas are written to handle the Alexa skill intents.

Advertisements

OAuth2 with Spring Boot and MongoDB

OAuth 2 is an authorization framework that enables applications to obtain access. This is token based authentication/authorization mechanism.

I tried to secure APIs by implementing OAuth2 with Spring Boot and MongoDB. MongoDB stores user details access and refresh tokens. To know more about the configurations visit my Github Repository.

7 qualities of an awesome developer / consultant

Having worked across many teams with some awesome developers, I could jot down some of their qualities. These are not all the qualities they have, but I think having these qualities certainly make you a good developer or a software consultant.

1] Attention to detail
This is such an important skill to have in a developer because producing error-free deliverable’s build trust with clients. This shows that you are thorough and your solutions are solid. Creating a check-list before pushing is one of the ways to not forget things.

2] Resourcefulness
You are not sure what kind of problems you are going to be asked to solve. Problems may be completely new to you – or even to industry. When you are faced with this kind of tough problems, it’s this skill that’ll help you succeed. You should be able to solve problems quickly and creatively.

You should be able to approach your lead, manager, your peers and resources outside. Being creative in research efforts is very much important. Be sure not to quit early and show your team/client that you can get the job done.

3] Not being afraid to ask questions
Asking questions to clients or stakeholders is at the core of how a developer works. You should feel like you are an interrogator :). It also helps in attention to detail.

Asking questions help you understand what you are being asked to do. Asking questions is better than doing something wrong. If you have understood the requirements or problems, rephrasing or restating those can inspire confidence with your client.

4] Not being a yes-man
Making the customer happy does not mean you have to agree with whatever they ask or tell. Developers are closer to business, technology and end users. You should be able to find the right balance between business needs, user needs, and technical feasibility. Guide your customer towards the right approach providing right reasons.

5] They listen to others
Building great software is not a one man job, it’s a teamwork. Learn to value everyone’s opinion. Accept the fact that it’s okay to be wrong sometimes. Learn to use appropriate tone while asking/requesting someone to do something.

6] They do not ignore the silent ones
You might have some people in your team who lack participation in decision making. It might be because the person is introvert, have conflicts with others or the person does not feel that conversation is of any value to him/her. Try to include them in the discussions, take them along. You never know, they might have something really valuable which can lead to a breakthrough.

7] They don’t get easily influenced
You might end up with a team which has one or more people who are very vocal and will force you to agree with the things they believe. They are not always right, so do not get carried away.

Install docker CE on Ubuntu 14.04 (Trusty)

Install Docker CE on Ubuntu 14.04 (Trusty)

1] Uninstall older versions of docker if any.
sudo apt-get remove docker docker-engine docker.io

2] Update the repository
sudo apt-get update

3] Install recommended extra packages for Trusty 14.04
sudo apt-get install \
linux-image-extra-$(uname -r) \
linux-image-extra-virtual

4] Update apt-get and install packages to allow apt to use a repository over HTTPS
sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common

5] Add docker GPG file
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

6] Verify fingerprint
sudo apt-key fingerprint 0EBFCD88

7] Use following commands to setup stable repository. You can also install builds from test and edge repositories.
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"

8] Update apt and install docker ce
sudo apt-get update
sudo apt-get install docker-ce

To test if docker has been installed correctly, execute the following command.
sudo docker run hello-world

To list all the docker local container:
docker ps -a

docker-ps

To list latest created container:
docker ps -l

To find out IP address, port and other information of Docker container use inspect command. This will print the JSON structure which contains all the information about the docker container.
docker inspect CONTAINER_ID

Spring Events

Spring Events

We will be discussing how to use events in Spring and will learn annotation-driven event listeners. In Spring, event publishing is the capability provided by ApplicationContext.

Few things to remember:
1] Event class should extend ApplicationEvent
2] Publisher class should have ApplicationEventPublisher object injected
3] Listener class should implement ApplicationListener interface

Consider a scenario where you have to trigger an email after any type of entity is created. An entity could be a user, project, organization, deal, task.
So the flow would be something like:
1] New entity has been created/saved in the database.
2] Create new entity created event which will be published by ApplicationEventPublisher.
3] ApplicationListener class will listen to the event and will create/send an email.

Suppose you are creating a Project. Project structure would like this:

Create Event class which extends ApplicationEvent. We are creating generic event class to handle multiple entities like projects, users, tasks etc.

Publish event after an entity is created.

Listen to the event and send mail after an entity is created.

Conclusion
In this tutorial we went over the basics of events in Spring – creating a simple, custom event, pushing it and then handling it in a listener. There is support for async processing as well. You learn more about Spring Events, visit this site.

Connection Pool Analysis

Why do we need connection pool?

I was working on a task where cron jobs were taking hours to complete, those were long running processes. To reduce the time of execution one way was to have worker threads to accomplish the jobs. I used ThreadPoolTaskExecutor and assigned the jobs, but came to know that after certain time application runs out of database connections. When I looked at the database configurations, found out we had DriverManagerDataSource data source which is not pooled data source. So to resolve this problem, decided to use a connection pool in the application.

The flow of database operation:

Database operation requires a connection. And the flow is like this:

1] Application asks the data source for a database connection.

2] Datasource uses the database driver to open a database connection.

3] Connection is created and TCP socket is opened.

4] Application performs the database operation.

5] Connection is closed and the socket is closed.

This involves opening and closing of connections which are expensive operations. This is extra strain on database management system to open / close connections.

So the best option is to reuse the database connections. This can reduce OS I/O overhead of opening/closing TCP connections.

Why pooling is useful?

Whenever a connection is requested, the data source will use the available connection pool to acquire a new connection. The pool will only create new connections when there are no connections available and a pool is not at maximum size. When the close() method is called on the connection, it’s not closed but returned to the connection pool. So reducing the overhead of creating/opening the new connections and making use of existing connections, connection pools are much faster.

Selection criteria:

Our criteria to decide which connection pool to go with.

1] Reliability

Performance is good, but it should not come at the cost of reliability. And we certainly think that reliability of library is important than performance. We don’t want to end up with the library which is faster than others but not reliable.

To decide the reliability of library, the following things we can consider:

  • How widely it is used
  • How is the code maintained
  • How active the library development is
  • Number of outstanding bugs in the library
  • Community of developers/users

2] Performance

The performance of the library again is really important criteria. The performance of libraries may vary depending upon how they are configured, the environment in which tests are carried out.

It is important to carry out the tests in the environment which you are selecting the library for.

3] Features

It is important to look at the features libraries provide. What are parameters they provide, how sensible are the default values of the parameters if we don’t provide them.

It’s important to look the connection strategies:

  • Do they reset connection properties when they return to the pool? E.g. auto-commit, isolation level
  • Do they support statement caching?

4] Ease of use

It’s also important to know how easy it is to configure (or to mis-configure) the connection pool using the library. It is important to have well documented/updated library.


Connection Pool Libraries considered for analysis:

1] Apache Commons DBCP2:

  • Easy to configure. Very detailed configuration page.
  • Supports prepared statement caching
  • Does not validate connections on borrow
  • Does not reset transaction isolation when returns to pool
  • Does not reset read only status of connection
  • Does not reset auto-commit status of connection
  • Does not clear SQL warnings
  • DBCP2 compiles and runs under Java 7 only (JDBC 4.1), not sure if it runs on Java 8
  • More than 60 classes in the library
  • Does not validate connections on borrow
  • The code is hosted on GitHub https://github.com/apache/commons-dbcp. The repository has 21 contributors and only 2 contributions in the year 2017. A lot of broken links on home page, java doc links are broken.
  • I logged into Apache Bugtracker (JIRA)  to check for the number of bugs, there are 11 open and reopened bugs.
  • Licensed under Apache 2.0

2] C3P0:

  • C3P0 is one of the oldest and best-known libraries. It is very well documented. However, according to some reports, it’s easy to misconfigure the pool and can result in poor performance and deadlocks.
  • Comes in package with hibernate.
  • Works well in the heavy concurrent environment.
  • Supports prepared statement pooling
  • Resets auto commit status when return to pool
  • Resets read the only status when return to pool
  • Resets isolation level when returning to pool
  • Does not test connections at getConnection()
  • Around 200 classes in the library. Huge code base compared to others.
  • The code is hosted on Github -> Repository, the library has 5 contributors with 5 contributions to the code this year, 4th May had the last check-in.
  • Issues are tracked on Github tracker, and has 41 issues listed.
  • Library is licensed under LGPL v.2.1 and EPL v.1.0

3] Tomcat JDBC:

  • Supports highly concurrent environments and multi core/CPU systems.
  • It is very well documented here.
  • Async connection retrieval. Can queue connection requests and return Future back.
  • Ability to configure custom interceptors to enhance functionality.
  • Provides multiple ways to configure pool. One is inside the Apache container, JMX and standalone by using DataSource bean.
  • Does not by default reset auto-commit and transaction levels for connections in the pool, users must configure custom interceptors to do this.
  • Does not by default test connections on getConnection()
  • Does not close abandoned open statements.
  • Pool prepared statement properties are not used.
  • The code is hosted on GitHub. It has 13 active contributors.
  • This is a very active community, one can see looking at the commits and bug list.
  • LLicensedunder Apache 2.0

4] HikariCP:

  • Very light library, around 130 kb.
  • Tests connections at getConnection()
  • Tracks and closes abandoned connections.
  • Clears connections before returning connection to client.
  • Resets auto-commit, transaction isolation, read-only status
  • Library has 58 active contributors and has very detailed configuration page with performance benchmarks.
  • Library has 33 issues listed on github
  • Licenced under Apache 2.0

Tests carried out with different pools:

The test results may vary with the configurations and environment they are conducted on.

I ran these tests on local developer environment with initial pool size of 10 and max size of 50.

1] I tried different connection pools and took some readings by calling database method 1000 times. Below is the table of results.

Pool & Readings (in ms) Reading 1 Reading 2 Reading 3
DBCP2 54242 48204 51760
C3P0 59204 60008 56360
HikariCP 49797 39998 46658
Tomcat CP 51156 49734 49734

 

2] Carried out tests by opening and closing the database connection thousand times. Below is the table of results.

Pool & Readings (in ms) Reading 1 Reading 2 Reading 3
DBCP2 455 359 209
C3P0 139 156 130
HikariCP 173 13 144
Tomcat CP 54 71 32

 

3] Took some readings on developer tool to monitor the response time of API (To fetch all organizations and their users).

  • Readings with Tomcat JDBC CP

pool_tomcat

  • Readings with DBCP2

pool_dbcp2

  • Readings with C3P0

pool_c3p0

  • Readings with HikariCP

pool_hikari2

Configurations used for performance testing:

1] Tomcat JDBC

2] DBCP2

3] C3P0

4] HikariCP

Final thoughts:

After going through all the resources, tomcat JDBC and HikariCP seem to be reliable and faster than other two connection pools. Both the communities are very active. Their codebase is smaller than other two. DBCP2 once was not active at one point of time but came out of dormancy later, not sure how reliable it is. Stars and forks on HikariCP and Tomcat repositories are more than that of other two connection pools.

Online resources:

http://www.trustiv.co.uk/2014/06/battle-connection-pools

https://blog.johnhagler.com/2016/08/13/pooling-mysql-jdbc-connections/

https://techblog.topdesk.com/coding/choosing-a-database-connection-pool/

https://vladmihalcea.com/2014/04/17/the-anatomy-of-connection-pooling/

https://www.scribd.com/document/111649850/Connection-Pool-Compare

http://brettwooldridge.github.io/HikariCP/ludicrous.html

https://github.com/brettwooldridge/HikariCP/wiki/%22My-benchmark-doesn%27t-show-a-difference.%22

https://github.com/brettwooldridge/HikariCP/wiki/Pool-Analysis

https://www.wix.engineering/single-post/how-does-hikaricp-compare-to-other-connection-pools

https://www.scribd.com/document/111649850/Connection-Pool-Compare

https://stackoverflow.com/questions/520585/connection-pooling-options-with-jdbc-dbcp-vs-c3p0

Prevent XSS security threat in spring application

What is XSS (Cross Site Scripting):

XSS (Cross-site Scripting) is one of the most common vulnerabilities with a web-application. And, it can be exploited by hackers very easily without using any sophisticated tool. Click here to know more about XSS.

How does it work?
Most web applications have forms to receive inputs from user. A form having text field can be exploited by a hacker.
A hacker can input javascript code e.g. or eval(“”) in the *userid* text field. When the form is
submitted, the server does process it & returns. This is not the scenario we want to happen.

Suppose your form looks like this (using spring tags):

The hacker may input something like ” jeevanalert(“I won.”) ”  in the text field. When the page is submitted, the page is returned with error as the user is not authenticated.

Solution:
If the input-text is changed into the html-escape-sequences during processing on server, then browser will receive
the escape-sequences. The resultant text would not be a valid java-script and browser will not execute it.

After conversion into html-escape-sequences, the above code would look like:

 
Since all instances of characters ‘<‘, ‘>’ and other non-alpha-numeric characters are converted into html-character-sequence, it is no longer a java-script code. So, it will not be executed by the browser.

Solution with Spring MVC In Spring-MVC, form-tags are used to create jsp page. Spring MVC provides multiple options
to encode the html-escape-sequences on server side.

  • At global level, it can be defined in web.xml file. This will be applicable to the entire application. All form-tags would refer
    to this definition. The sample code is shown below:
  • At page level, it is defined as a tag-declaration. The code is: Any form-tag, after the above declaration uses html-escape-sequence-encoding.
  • Third option is to define it as attribute for each form-tag. For example, a input-text can be defined as :

Depending upon the requirement, it can be implemented as global, page or tag level.

Filter Approach:
 Another solution to the XSS issue is to filter all the textfields in the form at the time of submitting the form. It needs XML entry in the web.xml file & two simple classes.

The code for the  first class named CrossScriptingFilter.java is:

The code second class named RequestWrapper.java is:

The only thing remained is the XML entry in the web.xml file:

The <url-pattern>/*</url-pattern> indicates that for every request made from browser, it will call CrossScriptingFilter  class. Which will parse all the components/elements came from the request & will replace all the javascript tags put by the hacker with empty string i.e. “”.

We have fours approaches to prevent the XSS threat.
You have to choose the approach according to your need.