web-dev-journal

Efficient Analytics on DynamoDB data

Everyone's talking about big data. I thought I'll talk about my experience too. Let me tell you this post is a bit long. I tried to keep it very short but failed. So, I have highlighted two sections which you could read and get the gist! (But I really want you to read all of it:)

Prologue: So, we at Simplilearn decided to capture some very arbitrary data from our applications. We thought, well, we need something that scales, supports very high throughput, low maintenance and of course, is cost effective. DynamoDB was right there. It's NoSQL so we were not worried about structure. It is a managed service so no worries about throughput and scalability either. We could also control the cost a little bit by adjusting throughput, but in spite of that, it costs us good money, so I won't give this one to DynamoDB.

We created a data store and started capturing data. There was some concern regarding queries we could run and the fact that reading data has a cost as well. The cost was something we could not afford to think about! So we focussed on the querying part. We had to move data from DynamoDB which is a datastore to a warehouse. That's how it works right, move data from datastore to data warehouse, but which one. And then there was revelation. Just kidding, we used Redshift this time.

Now, moving data from DynamoDB to Redshift is like just one command. And what that does is reads all data from a DynamoDB table and inserts into a Redshift table. That's the problem right there, reads. Remember I said, reading has a cost as well. Consider this, We are generating roughly 100GB of data every month and let's say we want to analyse our data weekly, we will be copying 100GB data after the first month. 125GB a week after that, 150GB one week after that ... eventually terabytes of data, every week. Now that can't happen!

So we needed a way to copy data incrementally. Enter, DynamoDB streams. How streams work is that you query DynamoDB with a timestamp and it gives you all the modifications since that time. So now we can have a daily cron job, that will get the stream since yesterday and insert/update data in Redshift. That done, we could just use all the column oriented goodness of Redshift to analyse our data. I'll post details about how to query some special cases like unix timestamps and JSON data in another post.

Epilogue: So there you have it. You don't use a datastore like DynamoDB for ad-hoc analytics queries. You use it's high throughput, availability and reliability in capturing mission critical data. You copy this data to a data warehouse like Redshift to run all the queries that the Analytics team keeps bugging you about. That way, you can also archive data month on month in DynamoDB and still have a historical dataset in Redshift.

Also, we are now evaluating Google's BigQuery. It supports a datatype called Record which basically is nested data that you would get from a NoSQL db. Also, its support for functions like date makes it very easy to work with my kind of data. I will discus that in an upcoming post. Till then, hang around!!!

Reasons why a you should go open source for your software's growth (part of Growth Hacking Series)

Is your software your primary USP?
Do you have a mature software platform that serves a specific purpose, developed by you and your team?
Do you have hundreds of thousands of people using your software?
Is your software under continuous development.

If the answer to all the above questions for you is yes, then here is why you must release your software source code under an appropriate license.

1. Limit to innovation and experiments over time
Developing a kick-ass product and doing it right is really a great achievement. I have seen many good ideas just not make it through the first 6 months. So if you have hundreds of thousands of users for your application, you have really achieved something. Have a champagne or something and celebrate. And after that, just think for a moment where your product will be heading from here. You can get a team to work on the product and figure out what new features you want to add. Maybe, you already have a team in place doing that, but the truth is, there is very little you will be able to achieve that way. You can conduct tens of experiments every month but users will hardly like some of them. Think about Google's Gmail, it was launched in April of 2004 and has taken 11 years to become what it is today. On the other hand take something like Android OS, again owned by Google. It is open source and has gathered a large community working to improve it. It has become very intuitive and easy to use in just 6 years. It is competing head on with Apple's iOS, which is considered to be a marvel in terms of usability. If you find a bug in Android and you think you can fix it, you can download their source code, fix the bug, create a patch and submit to them. This makes every Android user a potential QA and a developer. There are other ways you can contribute to Android by reporting bugs, etc. If you don't want to contribute, you can just release it as another OS. All this makes Android a very engaging experience which people flock to.

2. But there are so many open source software out there.
That is true, and here is where your hundreds of thousands of users are going to help you. When you release your software with an appropriate license, it makes it mandatory for people who are using it, to provide proper attribution to the creator of the software, in this case you. Checkout some popular blogs and look out for something like "Proudly Powered by Wordpress" or simply "Powered by Drupal". These blogs are using the said open source software and providing proper attribution to the authors of the software. Of course these can be removed but you can license your software so that it becomes mandatory. Now when a blog reader sees the attribution and finds out how easy it is to create his own blog website, he becomes a user too, thereby increasing the reach of the software. This is self driven growth. You did not have to invest, but you are getting more users and also becoming quite popular. And most of entrepreneurs will know the importance of this.

3. Is that secure enough?
The short answer is yes. Let me give you an example, you must have used some ssl enabled websites, the ones where the URL starts with https://. Well guess what, these servers are most likely using OpenSSL, an open source software that enables ssl over the internet connections. Every once in a while vulnerabilities are found in this software, but because OpenSSL has such a large community, that the bugs are quickly fixed and patches start circulating in a matter of hours. OpenSSL is being supported by Department of Defense and Department of Homeland Security. So the point being, if you do it right, you can have a very secure open source software.

4. There are many closed source products that have made it big without going open source
That is mostly true for either early products like Microsoft's Windows OS or exceptional ones. But if you look closely, there will be some parts of these products which are open source. Coming back to Microsoft's Windows OS, it is a very popular OS, but it hails from the days when things like open source did not matter. It has a very strong hold in the market, but it is really losing traction gradually. On the other hand, open source Linux based operating systems are gaining traction. Even if Slowly but surely. If you project statistics, you can clearly see where it is all going.

5. Do I have to release source for entire software?
No, you can release parts of it and keep some of the software to yourself. You can even have an alternate version which is open source and may not release your own software. Something like Google's Chrome browser. It has an alternate open source version called Chromium. You can contribute to Chromium but not to Chrome. You can even create your own browser using Chromium. In fact Google Chrome draws it's source code from Chromium. Sounds confusing? It's really not. In this case think about yourself as a vendor using the software. You would get the source code, customize the branding part and whatever is allowed by the license and use it as your own. So Chromium is the open source software here. Google is pulling code from it, making other changes and releasing it as Google Chrome. What they are giving a part of software and what they are receiving are a lot of contributions in terms of patches, bug reports etc. Compare this with Microsoft's Internet Explorer and you will know what I mean! In essence, you may release part of your software as open source and keep some of it proprietary to yourself. But it does not make sense here to have an open source part that will not work without the proprietary one. Your open source software should be a complete working product, even if it is with a few missing features. Even Facebook has many software which are open source, but their main platform is not.

To put this in final words for the article, you should seriously consider going open source with your well adapted software product. Going open sources create innumerable opportunities for the growth of your product. It will drive more users and create a community which is interested in that same area as your product. You can also leverage this community for many benefits ranging from accepting patches from them to getting suggestions for cool new features. The only key to do this right to be careful about choosing the license and keeping certain parts to yourself. There are experts available to help you with both of these.

What do you think about it? Let me know in the comments below...

Truth you need to know about SEO

Recently I was given a task from my department head. I was surprised when I got the details on what I had to accomplish for my organization's website. I am a hard core developer, not analyst or an SEO guy and I was told to increase website traffic. And that too by 4 times in three months!

Things started becoming clearer when I got into it. We formed a team. Myself and one more developer. Two guys with expert level knowledge in SEO and the HoD himself. The first step was to research. There were lots of topics to be researched on. Of course the SEO guys had lots of knowledge already. The research part was more geared towards me and the other developer.

We all met daily and shared progress and new information. This is when I realised that I had an important part to play in the whole exercise of getting more visitors organically.

There were lots of things coming up. I was on my computer day and night researching about stuff I had totally ignored before. Starting from ranking analysis to keyword research and all, terms like backlinks and link juice. When tasks were created from all the research, it was amazing how many of them couldn't have been done without coding.

We came to know that SEO actually has two types of tasks, technical and non technical. Technical tasks like adding proper canonical URLs to all pages of the site, proper and non repeating meta data, social integration and even marking up html with rdf attributes. And the non technical tasks were like keyword research, writing content, backlinking and using social platforms to drive more traffic in.

We started off by SEO guys running several crawlers on the website and finding out issues like duplicate content, missing title, meta description etc. It was a mess. The issues were fixed by technical guys. For all pages, we checked for a trailing slash in URL in our web app and redirected if required. We streamlined canonical urls where a subset of data was displayed from a larger set. We created a backend module for SEO guys where they could set title, h1tag on the page and meta data. They were able to mark some page with noindex,nofollow meta tag which had lot of duplicate pages. Then came the performance of the website. We concentrated a lot on the onload time of each page. In the end, we actually felt the quality of the site going up.

We were hoping to see some improvement, SEO wise, already but it would've taken another two weeks to figure out, so we continued with next task. At this time we had two tasks in mind. We can integrate more with social platforms and drive more traffic from there or we could mark up our html with proper rdf tags as specified in Schema.org. We selected RDF because we believed it would increase CTR (click through rate) from SERP (search engine result page) and that would drive considerable relevant traffic. We used Breadcrumb, Rating, Review, Event, Product, Organization and Video meta data to start with. Meanwhile we were already seeing some improvement in ranking because of the hygiene we had performed earlier. It was time to move on to next task.

Deeper social integration was still pending, but we decided to do a little experiment at this time. We were de-prioritizing social integration because we thought it may drive traffic, but the conversion rate for such traffic would be much lower. It is not relevant traffic. To bring in more relevant traffic we needed to cover more keywords, long tail keywords even and somehow indicate to the search engine that certain pages are product pages and are to be given more importance. So we created an RSS feed of all the products on the website and published it. We included lot of content from the blog in it. On all product pages, we added a rel=alternate pointing to the feed. We created the feed in such a way that everyday at 9am, fresh data is added to it. This was an experiment and we were not expecting earth shattering traffic from it.

Meanwhile, Google search engine started to show up rich snippets for breadcrumb, events and rating. We checked the CTR. It was going up. I was so excited to see measurable progress, I wanted more. 3 weeks had gone by and there was some visible progress. Me and the other developer now started with deeper social integration. Basic integration like sharing, like and login was already in place but we wanted to enhance it. We checked out how online marketing people were using our company's social pages. We went through the content they were posting and all and found out a couple of things we can do. For example, we should use more hashtags. Someone also suggested we should run weekly contests on social networks. Also, we decided to do something new with social platforms.

We saw all the available APIs provided by different platforms and decided to automatically create a new post on our page, whenever new products were added for a specific category or whenever new events of a specific type were scheduled. We used specific categories and types so that we could compare referral traffic for them with rest of the products and events. We also figured out some hashtags which we should always include. It took another week to implement this.

Within a month we had touched almost all the areas in SEO. With hygienic and high performance website, we had improved user experience. With RDF, we increased CTR considerably. We measured impressions for many long tail keywords going up and the RSS feed had improved that. We got considerably more social traffic by properly using hashtags in social posts and automating them. We started posting for all the products and events, but limited the overall posts we were creating. When we checked the stats, we had about 90% increase in organic search traffic from previous month and about 50% more social traffic.

But, it gets tougher from here. We have done a lot and still achieved about 35% of our target. We are back to researching phase.

What do you guys think, are we on right track?

PHP Unit testing with PHPUnit

Unit testing in PHP is easy and does not take much time. Below I have written a tutorial for introduction to Unit testing using PHPUnit.

Let's create a Math class with 2 methods viz. add and multiply. We will create this class as well as test cases for it. Actually first we will create the test case for add method and then the method itself. We'll repeat this for multiply method.

Creating the test case for Math::add():

Requirement: add takes 2 parameters and returns the sum of these two parameters

Test case

So we start with <?php

Then next we have require_once statement to our future class file.

Then we create a new class MathTest which extends PHPUnit_Framework_TestCase . This is very important.

Then we override 2 methods from parent class. For now I am leaving these empty and we'll come to this later.

Then we create the testAdd method. This will be the actual test. These “test cases” always begin with the word test.

Within this method we instantiate Math class. Create 2 variables to pass to add method as arguments and 1 variable that is the expected output.

The next line actually is simpler than it looks. It basically calls the Math::add method with 2 arguments and compares with expected output.

We've not created Math class, but I'll run this test case anyway just to see what happens.

The font in the image might be small but it says

PHP Warning: require_once(Math.php): failed to open stream: No such file or directory in /home/amit/unittest/MathTest.php on line 3

PHP Fatal error: require_once(): Failed opening required 'Math.php' (include_path='.:/usr/share/php:/usr/share/pear') in /home/amit/unittest/MathTest.php on line 3

Which means it could not find the class Math.

Now let's create the Math class with and add method.

This is a very simple class.

Now let's save this as Math.php along side the MathTest.php and run the test.

Notice there is a dot somewhere in the middle from top on left hand side. This is actually saying that the test was running. The final line says (that everything is) OK.

So this was the first test case. Again, just for demo, let's change the expected value to 6 but keep the arguments same so that our test case will fail.

When we run the test again, the output is something like:

See it says FAILURES at lots of places.

Now let's modify the test case again.

What I did is this time I changed expected value back to 5 and created a new variable $notexpected and set it to 6. Then I've used assertNotEquals to make sure that the result does not match $notexpected.

When we run the test.

1 test, 2 assertions. This means we can write as many assertions in a test case as we want. I'll go ahead and modify the test case again to demonstrate this further.

This time I've created 6 assertions inside one test case.

When we run this test:

Similarly let's develop the multiply method. First the test case.

Writing this test case took about 1 minute, though this is simple example, the most complicated one will also take not more than a few minutes, if you know what you are doing.

We can create more assertions inside the testAdd method but it is a good Idea to create a different test case for every method in your class.

Also you will notice that, for 2nd assertion I save the value that my new mthod will return to $result variable and use that to assert. And in 3rd assertion I've not created the expected and notexpected variables, but directly substituted their operations.

Now let's create the method.

Pretty simple. Now let's run the test.

Great, all tests passed. Notice the 2 dots instead of one. This indicates that it ran 2 tests.

You may think that to test a 3 line method, we have created such a long test case. Writing that test case actually took more lines than writing the program. But in reality, how many of your methods are 3 lines long? And instead of opening urls and waiting for the pages to load everytime we need to test, we can just run the test case which executes all the tests sequentially and tells us what problem there is.

Array functions in PHP - the way we do it - part 2...

RESTful web services api coding with PHP and CuRL

Hi,
I have had a chance to work on the RESTful web services offered by some of the giants in the internet industry, major ones being Rackspace, Yahoo and Google (especially with the custom search). It requires serious programming skills and is not at all for the faint-hearted. My experience involved downloading and uploading files with Rackspace cloud files api to Yahoo store partner management and even the custom search by Google.
RESTful api as the name suggests is basically transfer of state of server from and its representation. When you want to perform operations on a remote server, REST can provide maximum reliability and security. The trick is to represent the request and response in a way that the communication is easy and swift, but at the same time being reliable.
The requirement from the user endpoint to successfully carryout the operations on remote server is a formatted request sent to a specific location on the server. This location fires up a command on the remote server which performs the required operation. The user here is called client and the server is, well a server.
When the client performs a request, the REST protocol requires the request to have all the data available in a single request itself. So no temporary data is stored on the server, except for the temporary authentication token(which is usually present). So this makes the server highly reliable along with being easy to reach. So this suggests that even if we are performing same operation again and again on same place on server, we have to clearly specify the location and operation in every request.
For example I am performing an operation an operation on a directory of the server through my computer, this operation will change the state of the server. So I create a request, send to appropriate directory on the server, the server is now initiated and operation is performed, at this time the client is considered to be in transition, then the server completes the operation and sends the response back. This might look new to some, but its basically the exact process that happens every time you start browsing any website. For example if you type http://www.google.com/ in your browser and hit enter, a request is created by your browser and sent to the root directory at google.com which then performs the operation of generating a webpage and sends the response back. This response usually contains head (headers) and body (content). The headers are usually for the browsers own use while the body is displayed to the user.
Another good example would be when you accept a friend request on your social networking site. What happens underneath the browser, when you click that button, is that your browser creates a request and sends it to a specific directory on the server. The server carries out the operation of updating the database and sending a notification to the other user, that the friend request is accepted, and returns a response to the browser. The browser displays the received response to the user. If the same process had to happen again, the REST constrains that each piece on information exchanged in the first time should be exchanged again.
This gives rise to endless possibilities like exchanging text or even binary data. To carry out this same process in PHP to accomplish some major tasks over remote servers, we need a way to send requests and receive response. CuRL library for PHP does the trick here. We can generate the request in the form of string or even binary data (if required to be uploaded to the server) and send it over HTTP. HTTP offers several methods like GET, POST, PUT and DELETE. These methods are highly useful for distinguishing the kind of operation to be performed on the server.
The Yahoo store api takes XML formatted data, while Rackspace cloud understands JSON and XML. There are major differences between the services offered by each of the providers and so the requests are also different. This makes the possibility of creating a reusable code very difficult.

To be continued... visit back soon.

--
Amit Kriplani
PHP Web/Application Developer
Blog:http://blog.amitkriplani.com
Mobile: +91-9913932644MSN: amitk87@hotmail.com
Gmail: amitkriplani5@gmail.com
Yahoo: amitkriplani@ymail.com
Skype: amitkriplani
Twitter: @amitk87

Array functions in PHP - the way we do it - part 1

Hi,
Back again! If you have read my previous posts then you will be sure this post is about PHP - the programming language that drives most major websites, at least more than 50% of them. Well we have already discussed how arrays are addressed in PHP. Now let's see how to do things with arrays without any panic and fear in our mind. Let me tell you PHP has a good amount of functions to manipulate arrays in all sorts of ways. This post will be as long as the great wall if I keep them all here. So, as they say, Break-it-down, I will break this post in several parts. First lets have a look at the functions available in PHP that can be used with arrays.

I found this list on php.net

array_change_key_case — Changes all keys in an array
array_chunk — Split an array into chunks
array_combine — Creates an array by using one array for keys and another for its values
array_count_values — Counts all the values of an array
array_diff_assoc — Computes the difference of arrays with additional index check
array_diff_key — Computes the difference of arrays using keys for comparison
array_diff_uassoc — Computes the difference of arrays with additional index check which is performed by a user supplied callback function
array_diff_ukey — Computes the difference of arrays using a callback function on the keys for comparison
array_diff — Computes the difference of arrays
array_fill_keys — Fill an array with values, specifying keys
array_fill — Fill an array with values
array_filter — Filters elements of an array using a callback function
array_flip — Exchanges all keys with their associated values in an array
array_intersect_assoc — Computes the intersection of arrays with additional index check
array_intersect_key — Computes the intersection of arrays using keys for comparison
array_intersect_uassoc — Computes the intersection of arrays with additional index check, compares indexes by a callback function
array_intersect_ukey — Computes the intersection of arrays using a callback function on the keys for comparison
array_intersect — Computes the intersection of arrays
array_key_exists — Checks if the given key or index exists in the array
array_keys — Return all the keys or a subset of the keys of an array
array_map — Applies the callback to the elements of the given arrays
array_merge_recursive — Merge two or more arrays recursively
array_merge — Merge one or more arrays
array_multisort — Sort multiple or multi-dimensional arrays
array_pad — Pad array to the specified length with a value
array_pop — Pop the element off the end of array
array_product — Calculate the product of values in an array
array_push — Push one or more elements onto the end of array
array_rand — Pick one or more random entries out of an array
array_reduce — Iteratively reduce the array to a single value using a callback function
array_replace_recursive — Replaces elements from passed arrays into the first array recursively
array_replace — Replaces elements from passed arrays into the first array
array_reverse — Return an array with elements in reverse order
array_search — Searches the array for a given value and returns the corresponding key if successful
array_shift — Shift an element off the beginning of array
array_slice — Extract a slice of the array
array_splice — Remove a portion of the array and replace it with something else
array_sum — Calculate the sum of values in an array
array_udiff_assoc — Computes the difference of arrays with additional index check, compares data by a callback function
array_udiff_uassoc — Computes the difference of arrays with additional index check, compares data and indexes by a callback function
array_udiff — Computes the difference of arrays by using a callback function for data comparison
array_uintersect_assoc — Computes the intersection of arrays with additional index check, compares data by a callback function
array_uintersect_uassoc — Computes the intersection of arrays with additional index check, compares data and indexes by a callback functions
array_uintersect — Computes the intersection of arrays, compares data by a callback function
array_unique — Removes duplicate values from an array
array_unshift — Prepend one or more elements to the beginning of an array
array_values — Return all the values of an array
array_walk_recursive — Apply a user function recursively to every member of an array
array_walk — Apply a user function to every member of an array
array — Create an array
arsort — Sort an array in reverse order and maintain index association
asort — Sort an array and maintain index association
compact — Create array containing variables and their values
count — Count all elements in an array, or something in an object
current — Return the current element in an array
each — Return the current key and value pair from an array and advance the array cursor
end — Set the internal pointer of an array to its last element
extract — Import variables into the current symbol table from an array
in_array — Checks if a value exists in an array
key — Fetch a key from an array
krsort — Sort an array by key in reverse order
ksort — Sort an array by key
list — Assign variables as if they were an array
natcasesort — Sort an array using a case insensitive "natural order" algorithm
natsort — Sort an array using a "natural order" algorithm
next — Advance the internal array pointer of an array
pos — Alias of current
prev — Rewind the internal array pointer
range — Create an array containing a range of elements
reset — Set the internal pointer of an array to its first element
rsort — Sort an array in reverse order
shuffle — Shuffle an array
sizeof — Alias of count
sort — Sort an array
uasort — Sort an array with a user-defined comparison function and maintain index association
uksort — Sort an array by keys using a user-defined comparison function
usort — Sort an array by values using a user-defined comparison function

Hmm! that's a long list. I promise all those are for manipulating or reading arrays.Yes there are 77 of them. Apart from these there are functions that return arrays of convert them to strings, like explode(), implode(), respectively. I will cover most of them but I will club the linked functions together. If you want a specific function to be explained in detail you can always leave a request in comments.
OK, so let's see at the functions individually:
1) array_change_key_case - this function changes the character case of keys in a associative array. Means if you call it on a certain associative array, it will return the same array but with all keys changed to lower/upper case. If the keys have mixed lower and upper case characters then this function returns ...

--
Amit Kriplani
PHP Web/Application Developer
Blog:http://blog.amitkriplani.com
Mobile: +91-9913932644MSN: amitk87@hotmail.com
Gmail: amitkriplani5@gmail.com
Yahoo: amitkriplani@ymail.com
Skype: amitkriplani
Twitter: @amitk87