Monday, March 23, 2015

Awk :: Log Processing Techniques - Web Server Logs

Re-posted from http://awkaslanguage.blogspot.in please visit and find more interesting tailored made awk training material.

As we have learnt some basics on AWK, lets proceed on applications like log processing. In this session we will see how to process webserver log files. Below is one example log file which we will use for our discussion.

Sample log file - download here
213.60.233.243 - - [25/May/2004:00:17:09 +1200] "GET /internet/index.html HTTP/1.1" 200 6792 "http://www.mediacollege.com/video/streaming/http.html" "Mozilla/5.0 (X11; U; Linux i686; es-ES; rv:1.6) Gecko/20040413 Debian/1.6-5"
151.44.15.252 - - [25/May/2004:00:17:20 +1200] "GET /cgi-bin/forum/commentary.pl/noframes/read/209 HTTP/1.1" 200 6863 "http://search.virgilio.it/search/cgi/search.cgi?qs=download+video+illegal+Berg&lr=&dom=s&offset=0&hits=10&switch=0&f=us" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"
151.44.15.252 - - [25/May/2004:00:17:21 +1200] "GET /js/common.js HTTP/1.1" 200 2263 "http://www.mediacollege.com/cgi-bin/forum/commentary.pl/noframes/read/209" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"
151.44.15.252 - - [25/May/2004:00:17:21 +1200] "GET /css/common.css HTTP/1.1" 200 6123 "http://www.mediacollege.com/cgi-bin/forum/commentary.pl/noframes/read/209" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"
151.44.15.252 - - [25/May/2004:00:17:21 +1200] "GET /images/navigation/home1.gif HTTP/1.1" 200 2735 "http://www.mediacollege.com/cgi-bin/forum/commentary.pl/noframes/read/209" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"
151.44.15.252 - - [25/May/2004:00:17:21 +1200] "GET /data/zookeeper/ico-100.gif HTTP/1.1" 200 196 "http://www.mediacollege.com/cgi-bin/forum/commentary.pl/noframes/read/209" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"
151.44.15.252 - - [25/May/2004:00:17:22 +1200] "GET /adsense-alternate.html HTTP/1.1" 200 887 "http://www.mediacollege.com/cgi-bin/forum/commentary.pl/noframes/read/209" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"
151.44.15.252 - - [25/May/2004:00:17:39 +1200] "GET /data/zookeeper/status.html HTTP/1.1" 200 4195 "http://www.mediacollege.com/cgi-bin/forum/commentary.pl/noframes/read/209" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"



1) To get all requests from a particular webpage

151.44.15.252 - - [25/May/2004:00:17:20 +1200] "GET /cgi-bin/forum/commentary.pl/noframes/read/209 HTTP/1.1" 200 6863 "http://search.virgilio.it/search/cgi/search.cgi?qs=download+video+illegal+Berg&lr=&dom=s&offset=0&hits=10&switch=0&f=us" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"

 
2) To find number of hits from each IP address

213.60.233.243 has hit webserver 1 times
151.44.15.252 has hit webserver 7 times


3) To find all the requests within a time period(assuming requests are ordered sequentially by time)

213.60.233.243 - - [25/May/2004:00:17:09 +1200] "GET /internet/index.html HTTP/1.1" 200 6792 "http://www.mediacollege.com/video/streaming/http.html" "Mozilla/5.0 (X11; U; Linux i686; es-ES; rv:1.6) Gecko/20040413 Debian/1.6-5"
151.44.15.252 - - [25/May/2004:00:17:20 +1200] "GET /cgi-bin/forum/commentary.pl/noframes/read/209 HTTP/1.1" 200 6863 "http://search.virgilio.it/search/cgi/search.cgi?qs=download+video+illegal+Berg&lr=&dom=s&offset=0&hits=10&switch=0&f=us" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"


4) To find number of hits each URL received

URL /adsense-alternate.html got hit 1 times
URL /internet/index.html got hit 1 times
URL /cgi-bin/forum/commentary.pl/noframes/read/209 got hit 1 times
URL /data/zookeeper/status.html got hit 1 times
URL /css/common.css got hit 1 times
URL /images/navigation/home1.gif got hit 1 times
URL /data/zookeeper/ico-100.gif got hit 1 times
URL /js/common.js got hit 1 times


5) To group all the requests from a particular IP address

213.60.233.243=>
213.60.233.243 - - [25/May/2004:00:17:09 +1200] "GET /internet/index.html HTTP/1.1" 200 6792 "http://www.mediacollege.com/video/streaming/http.html" "Mozilla/5.0 (X11; U; Linux i686; es-ES; rv:1.6) Gecko/20040413 Debian/1.6-5"

 151.44.15.252=>
151.44.15.252 - - [25/May/2004:00:17:20 +1200] "GET /cgi-bin/forum/commentary.pl/noframes/read/209 HTTP/1.1" 200 6863 "http://search.virgilio.it/search/cgi/search.cgi?qs=download+video+illegal+Berg&lr=&dom=s&offset=0&hits=10&switch=0&f=us" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"
151.44.15.252 - - [25/May/2004:00:17:21 +1200] "GET /js/common.js HTTP/1.1" 200 2263 "http://www.mediacollege.com/cgi-bin/forum/commentary.pl/noframes/read/209" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"
151.44.15.252 - - [25/May/2004:00:17:21 +1200] "GET /css/common.css HTTP/1.1" 200 6123 "http://www.mediacollege.com/cgi-bin/forum/commentary.pl/noframes/read/209" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"
151.44.15.252 - - [25/May/2004:00:17:21 +1200] "GET /images/navigation/home1.gif HTTP/1.1" 200 2735 "http://www.mediacollege.com/cgi-bin/forum/commentary.pl/noframes/read/209" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"
151.44.15.252 - - [25/May/2004:00:17:21 +1200] "GET /data/zookeeper/ico-100.gif HTTP/1.1" 200 196 "http://www.mediacollege.com/cgi-bin/forum/commentary.pl/noframes/read/209" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"
151.44.15.252 - - [25/May/2004:00:17:22 +1200] "GET /adsense-alternate.html HTTP/1.1" 200 887 "http://www.mediacollege.com/cgi-bin/forum/commentary.pl/noframes/read/209" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"
151.44.15.252 - - [25/May/2004:00:17:39 +1200] "GET /data/zookeeper/status.html HTTP/1.1" 200 4195 "http://www.mediacollege.com/cgi-bin/forum/commentary.pl/noframes/read/209" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.4.7.0)"

Tuesday, January 14, 2014

Cashew Bite

We have got two Capricorns and one Aquarius in the family. So apart from the 3 birthdays there's Christmas, New year and Pongal to pack December-January with celebrations. So I've got another excuse to put on weight.

Today my wife bought a box of sweet, Cashew bite from Sri Krishna Sweets. I'm not a big fan of Cashews unless it's a Saturday (pun intended)! This box of Cashew bite is as good as it can get. It's not too sweet and children will love it for a quick bite. Plus the whole nut inside gives a twist with each bite. It costs Rs 400/- for a box and has a shelf life of 25 days. Each bar contains Chocolate, wallnuts, badam, cashews, ghee, pista and sugar. I'm not sure of the calorific value. I'd highly recommend to try it!



It comes packed in a nice box.

 A quick shout out to Nikita Containers for the neat work.

  It's hard to resist opening the box, when you see it.


It's wrapped in a double layered foil, giving it a rich look.


And with all that, you have one family of happy customers. :-)

P.S. I'm a photographer at Light Story. This post is not a sponsored one. 

Thursday, January 2, 2014

A picture that speaks more than 1000 words


Recently I started to travel lite. Especially when Proton is around, I make sure it a thumb rule. This time I carried a single camera body and lens with a flash. My wife was able to squeeze 4 days of leave (saturday and sunday inc) and we spent quality time in Coimbatore. More than us, the grandparents and the extended family got a chance to spend time with Proton.


People at my home set high standard photographs from me. Even there was an agenda to hire a more popular photographer to get a baby shoot done. ;-) Some perfectly timed shot are rated as below average. So I need to keep pushing myself to get those perfect shots. Today I was reviewing the pictures from the trip and this frame struck me (not a technically sound image), it tells endless stories of how people's life is affected by little things. The smile on Proton's grandfathers and Mom says the mood. I'm able to see how the style of wedding photography I do has made me see the world differently. I love my job.

Happy New Year, 2014.

Wednesday, September 11, 2013

Cat, dog & roosters

No, its not the once in a blue moon update post from me.
Recently I'm having more time to look at life at its natural best. With the nativity of my inlaws to a country side life we decided to get some pets at home. Primarily its easier to baby sit my daughter with them around. Thanks to the PWD engineer who decided to build this govt quarters with a lot of real estate space for garden and a small house in the centre.

We have a bunch of squirrels running in the garden eating the seasonal fruits. They are busy and fast.


Then we got 9 hen, 1 rooster and 2 Guinea type. Then patrol the house 250 times a day looking for insects, ants and other food wastes. A recurring alarm early in the morning, so I have literally sound proofed the bedroom.


With the new arrivals in the compound, the old god father cat from the neighborhood is annoyed. He stops by and has even started marking territory!



But the rooster and gang are determined to make a statement "இது எங்க area, உள்ள வராதேய்" [English: Its our area, don't come here] until the cat walks away annoyed.


Last but not least, Mac. A stray dog we adopted from a busy highway & my daughters favourite so far. Sadly we lost Mac to a severe viral fever. May his soul rest in peace in the beautiful garden.


Update:
I have to update this post, as the latest pet that interests Proton is Night Fury! I knew pets are at times downside, but this is extreme. Someone bring home a Night Fury (to be exact toothless).


Wednesday, May 16, 2012

My interview with SupportBiz


Software engineer Vijesh has always extremely passionate about photography, especially candid shots. He has a totally different angle of looking at the mundane things of everyday life, and that is through his camera. A little over a year back, Vijesh decided to convert his passion for photography into a business with Light Story.

How long has Light Story been in operation?
I have been covering weddings for a long time, but Light Story as a brand been in existence for a little over a year now...

Read the full interview here http://www.supportbiz.com/articles/top-story/conversation-vijesh-light-story.html

Saturday, February 4, 2012

Story of my Wedding ring

In Dec 2009, I was busy shopping for my Wedding. I had got the suits and dhoti's I needed. Only thing that was pending was the Wedding ring. I'm not a fan of gold or jewellary but decided that I will be wearing the wedding ring more often. My requirements were, I wanted to get a simple circular ring and wanted to emboss characters in it.


We started the hunt in Chennai. Most shops here did have a circular ring but did not have custom character embossing in them. Few of them were only able to carve the name in the ring, that was machine done so only standard character fonts were available. Finally my mom suggested to try Kirthilal, Coimbatore. After speaking with the sales guy there I was introduced to an elderly man. Soon the sales guy started gesturing that I wanted a circular ring. Yes, the elderly man was physically challenged (he could not speak and hear), though he had excellent lip reading skills and a god's hand in drawing. He has worked as a jewellary designer close to 18 years. I then gestured to him what my requirements were, he then drew a three dimension of the ring alone. Whola! Perfect. I was then asked to write the text that I wanted in it, I wrote them in small case. He then made small touch up and redrew the ring again in 3d with the name in it. That was all, my ring got delivered precisely as I had asked for.

Well, that was the story of my ring. What is yours?
It was our second wedding Anniversary! Happy Anniversary to us! :-)

Wednesday, December 21, 2011

விளம்பர இடைவேளை

Ads are inseparable part of India media. From serials my mom watches to the boring formula one race that no one watches in my home except me there are ads and ad breaks. So there is lot of importance that's stressed in the ad industry in recent times, they are sometimes given more weightage than mainstream. Here was an post by paramesh that gets you to the time of DD.

I picked one best, one worst and one all time favorite ad in recent times.

Best one: Kanjoos friend Airtel. I liked it for the fact, it was short & sweet and conveyed the message.

Worst one: Lalitha Jewellery. Watch it for yourself and I got no comments. Sad point is I get to watch it in cinema screen every time I visit PVR.


Favorite one: The Force: Volkswagen. No wonder its in the most watched videos of youtube.


If you have got your collection, scribble it in the comments. I'm interested to see them.