Office, programming

A Ning Network + RSS Feeds + PHP cUrl + MySql + GAE -> WTF?!

We use Ning in our company for the developer community activities around our product.  We wanted to know if there are any topics on forums are unanswered, or what are the topics that were being discussed in the last week, or were any blogs written, and things like that.  To make it simple and easy to understand, something like what happened in the past week / month.

Googled to see if anything as such is available,  as my luck would have, i didn’t find any.  So, i decided to write my own.  Went to Ning API documentation to see if there are any ways to get the data, again a dead end.  It is not available as of now.  But then, i remembered seeing a rss feed of activities in the site, so decided to dig a little more into it.  and land up at this page.  Armed with this piece of info, & since this kind of stuff can’t be done as of now, with our product (if only they provided the data as SOAP service), i sat down to write something to parse the feed, read the data, get the required ones, store it and do the reports.

So sat down to write the feed parser.  decided to use the hosting which i signed up some time back, which i never used, for this purpose. Used PHP cUrl to read the feeds and process it and store it into mysql tables.  everything went well, until they blocked cUrl and socket programming.   I was not in a mood to search for another hosting which supported this.  also, this required me running the php page in a browser so that it can refresh every 15 minutes and get the new data.    So decided to look into something with scheduling and such.

When doing something similar at work, the idea of using Google App Engine for this dawned.  But then App Engine supported only python and Java, which would mean porting from php to any of these languages.  But then, it was Java which gave me a ray of hope.  Since there are many JVM compatible languages around, and if Java was supported, indirectly these languages should also supported.  after a little googling got this information.  And there it was, PHP will work via Quercus.  Digging around their forums, and after looking around, decided to go about doing it.  If it comes up well, then Good, else, it would be one hellava ride, knowing App Engine, Quercus and stuff.

And Challenging it was.  though everything worked without major changes, except storing of data (i was not understanding the examples on how to do it), there was some problems cropping up.  First fetching of atom feeds was not working.  something was a miss.  going around making changes, i couldn’t wat fixed it, after a day it started to work.  But still there was some problem with getting the data, the feed parsing was not returning the feed data for certain tags, turned it did not return the description tag’s data in the rss feed (it was wrapped as CDATA, and was working inconsistently).  Couldn’t figure out why.  and since was pressed for time, decided to fight it another day, and scrapped the required data from another tag’s data.

Next it was Task Queues.  It was not a big difficulty.  actually it was the most easiest, i had a working example.

At this stage, i was at the verge of losing my patience.  It had taken close to 2 weeks, and i was still fighting ( i was on it only on weekends, and you can imagine, how the distractions will be. Cartoons, Movies, Parents, Cooking, Going Out, Cricket, Friends, Twitter, Facebook, Sleeping, Caring for the bike, etc……).  i was pretty much tired of making changes, and just wanted something which will complete the work. Instead of using Data Store for storing the data, and then trying to figure out how to deal with aggregation and joins, decided to go with the mysql in my hosting.  so whipped out a quick script to send data to my hosting from GAE, a php page on my hosting to accept the data via post, validate it and store it into mysql, and another to display the information as  i had wanted it.

At last i have it, something that does what i wanted.  It was a kind of a bit messed up solution, which i had cooked up with all sorts of nonsense ideas ( it seems nonsense when i first thought of ’em ) which i came up.

Now need to, convert it to make use of datastore to store data, which is a trivial task, but then, to do the reports, i think i should change a lot of things, possibly the datamodel structure which i am using now, since datastore does not support joins, it is going to be a challenge for me.  ( i am still the same person, who is not comfortable writing any queries other than select * from table where… and delete from table where….  )

Like i said at the beginning, If it comes up well, then Good, else, it would be one hellava ride.  but frankly, it was both, and boy what a lot of fun it was………  : )

Links where i looked when i was stuck.

The first three are essentially the same…   I would love to share what i had done.  I will, after completing the datastore part of it.

Meetup

For those of you who missed chennai #techmeet 2 #chennaigeeks

The Second Techmeet of Chennai Geeks happened today.  And like, the previous session, i recorded this session too.  For those who missed, here are the links of the recordings.

Web Scale Computing by Mani Doraisamy Part 1 [107 mb], Part 2 [52 mb]

MemCached by Sudar Part 1 [105 mb], Part 2 [59 mb]

Meetup

For those of you who missed chennai #techmeet #chennaigeeks

After a late decision to attend techmeet last night, was feeling bad for a few of my friends who felt bad about having to miss it.  So i decided to help them the only way that i can,  record it.  So here i am posting the links to the files, for the general public as well.

Introduction of TechMeet by Dorait    Download [8.8mb]

Gist of Cloud by Dinesh Varadarajan   Part 1 [117mb],  Part 2 [12.7mb]

MongoDB by Vagmi Mudumbai   Part 1 [127.1mb],  Part 2 [81.6mb]

I hope the presentations too will be posted by the speakers.

Warning:  The audio quality is not that great.  (I was not in the right position, and my corby pro was a lill ill, and i really need to freshen up my memory on how to use Audacity).   Hearing on external speakers is ok.  But with headphones, may cause headache.  If some one could do some sort of cleanup, it would of great.

Competition, Office

How i stopped worrying and added a chart

Ok.. Now everything is over.  Ok, it was over last week itself.  I even got a iPod shuffle!  I will let you all see what i was talking about that night.

Bogzilla

Nice isn’t it?!  Yeah, I know.  Many were not even wondering about how i got that ui.  (Seems they all know how to do it, it was not a secret afterall!).  They were only asking about the pie chart.  So lemme go about how i stopped worrying and added the chart.

Well actually it was not a big thing.  I used Google Charts for generating the chart image and then hooked it up into my form.  That is it.   Ok, i can understand you are starting to frown.  I will give you the details.    Basically i got all the details that i required for generating the charts using the DCOUNTA() function.  That is what appears in the right column above the chart.

B8=DCOUNTA(Issue.CreatedBy,Issue.CurrentStatus,Issue.CurrentStatus=”New”,TRUE)

D8=DCOUNTA(Issue.CreatedBy,Issue.CurrentStatus,Issue.CurrentStatus=”Open”,TRUE)

F8=DCOUNTA(Issue.CreatedBy,Issue.CurrentStatus,Issue.CurrentStatus=”Resolved”,TRUE)

H8=DCOUNTA(Issue.CreatedBy,Issue.CurrentStatus,Issue.CurrentStatus=”ReOpen”,TRUE)

J8=DCOUNTA(Issue.CreatedBy,Issue.CurrentStatus,Issue.CurrentStatus=”Verified”,TRUE)

Now i had got the url constructed for this chart using the CONCATENATE() function.

=CONCATENATE(“<img src=’http://chart.apis.google.com/chart?chs=200×300&cht=p3&chco=93CE11|DF6800|FFEFCB|FF0000|CCCCCC&chd=t:”, B8,”,”,D8,”,”,F8,”,”,H8,”,”,J8,”&chdl=New|Open|Resolved|ReOpen|Verified&chdlp=b’ width=’200′ height=’300′ alt=” />”)

Ok, i had also added the HTML tags to display the image along with this… Sheeh!  is it that bad to bend a little, afterall how will i ever let the form know that the output of this an image, if am not going to add HTML img tag.

Click here to see the same chart which you are seeing in the screen shot.  You can look at the address bar to see the constructed URL.

You can find the explanation for the url parameters here.  If you find it too daunting to go thru and read all the info there, use the Chart Wizard to construct the url and then change the values with your field names or variables.

Have fun!

Office

Postponed!

The Review of mine and my colleagues has been postponed to wednesday!   Well,  i was kinda worried with what more i can do, and by evening, another idea appeared, which was kinda crazy, one question led to another, siva saying not possible, adhi saying no way, pavitha saying why doing all these nonsense, it is way more easy, in the midst of the chaos, it started to seem that, the crazy idea might be possible.  This can’t be disclosed as of now.  Siva was not sure if it can be done.

After nearly 3 hours with the paper mock ups and nothing leading nowhere, i decided to start with the first part.  after an hour and 22 minutes i saw the line.  The very first line which i saw when i started on this small adventure.  Yup!  it was right there in front of my eyes and i was searching for it for such a long time.

So now, I am sure, i can do this thing that i am planing, and if possible, i will do it by wednesday afternoon, so that i can show it up during the review, instead of the one that i am having now.

Since i have put tooooooo much suspense, i will give you this small clue.  Everything is a Form, I mean Everything…..  ok, that is not entierly true, i will use the other perspective’s also.

Competition, Office

“Create New” Wizard in Tracker

Ha! Atlast, i just completed the screen design and the permissions, for the Wizard to submit a issue, in the application, as i had decided with the mockup. It is not too great, but yeah, that’s a great achievement for me. After having had to handle everyone in the world, i got to complete this., Wheeeeee! ok. i need to cool down. so this is it….

Yes!, The design has been done, and it is working good. No problems yet!, (coming to think of it, i have not yet submitted from the final screen. Wait, ok even that works). It flows good, moves from one screen to another normally, the help messages come up good. yeah everything fine, except the images loading times, coz the images have to come from my storage on the internet instead of from application, it is taking time. But other than that everything is fine.

Now i need to think of other ways to make the other screens in the application, interesting.

Something i noted in the product today, was that the deployment popup was retaining the Applicaiton Id and UserId for authentication most of the time, but yeah few times it was not there.  I don’t know why.  Have they implemented this feature, or is it Firefox playing tricks?!

Competition, Office

“Create New” screen Mock Up

I hope you guys know of the Developer Competition which just recently ended up last week.  I had been developing an app for it.  But after seeing the previews on Friday Evening, i know one thing.  I don’t stand a chance of winning.  So, now i am going to do one thing which is the sensible thing.  The Standards have been raised, so now i have raise to that level.

The one thing which i was developing is an issue / bug tracker.  I have created a mock up for the issue creation screen, which i am going to sit thru and create today.  Thought i would put it up here, so i don’t have to refer for it anywhere else when required.

This is how the screen is going to look.  I am changing it to look like a wizard, making the user to go thru steps to collect the data regarding the issues.  I am still in confusion as to how to use put the step numbers, either as just numbers in a square box, or like the one which is drawn at the top, seperately.  Will put up a screen shot as soon as i am finished with it.,

Office

ostechtalk Procedural Programming vs Rule based Programming

Every friday for the past month we have been having debates among the knowledge sharing teams in orangescape.  This last friday (24th Sept 2010) was ours.  The debate topic was Procedural programming vs Rule Engine based programming.  Me & @sivaacbe represented Procedural programming.  @balajivg and @vimalkumar_v represented Ruled Based Programming.   The essence of what we (Procedural Programming)  spoke during the presentation phase is going to be content of the post.  you can follow our team’s progress here.  you can check #ostechtalk in twitter

Both Procedural as well as rule based programming have their own advantages.  As the advocates of procedural programming we started of with the explanation of the evolution of languages.  Instead of talking the usual generations and stuff, we went with a different approach based on the decades.  starting the with the birth in the 40’s, to the innovations of the 60’s and enhancements of 70’s and 80’s and closing with the productivity empasis of the 90’s and 00’s, and continuing with the future of programming languages.

What do we have against Rule Engines?

Actually speaking, nothing.  Rule Engine based Programming which comes under the logical programming paradigm may has its own advantages.  but they do have their own problems.

  • First of all it is just a small piece of software in the large picture.  How many times are we going to need a Rule Engine.
  • They are powerful.  But they are mighty powerful, that many a times using it becomes a overkill, unless and until you have a problem that can only be solved by Rule Engine, using it is not advised.
  • Rule Based System do not know how to handle maybe conditions.
  • It is a new learning, and sometimes the learning curve might be high.
  • Testing the rules alone is difficult.  and also you can never be sure which other rules are going to get affected because of this rule change.
  • There are no standards for Rule Engines.

This is the transcript of what i spoke with the presentation.

Evolution of Programming Languages

A programming language is an artificial language designed to express computations that can be performed by a machine, particularly a computer.

-Wikipedia

Though Wikipedia says a programming language is a language to express things to a computer, the existence of programming languages predates the birth of computers (Yes, the one on which you are reading, and not the generic computers).

  • The profession of programming is traced back to the mid 1800’s.
  • In the early days of computing, there was no difference between the software and hardware.
  • Programs were written directly into the hardware.
  • Often the Hardware restrictions defined the language.

Inception

  • The first general purpose computers started to appear during the 1940’s.
  • These Computers were capable of storing and loading the programs.
  • Developers used machine code to program these Computers.  Machine Code is nothing but 1’s and 0’s.
  • Still these were difficult to write programs, and even more difficult to debug and fix errors.
  • These were the First Generation Languages.
  • Soon these were replaced by mnemonic codes (Assembly Language) which were slightly easier to code.
  • Still these too were difficult to debug.  These were the Second Generation Languages.

Compilers

  • In the 1950’s the first modern programming languages came into being.
  • The Big Daddies of all programming languages which exist today – FORTRAN, Cobol were created in this period.
  • There was a move towards abstraction of complexities and making the language as easy as possible for the human to understand.
  • The programs written in these languages were converted to Assembly or Machine Code using Compilers, which led to this era being named as the Compiler Era.  This led to the wide spread adoption of these High Level Languages.
  • The Third Generation of the programming Languages began.
  • Even at this stage of evolution, the language capabilities diverged as to solve specific programming problems.

Paradigms

  • Software was beginning to become bigger and complex, during 1960’s and 1970’s.
  • Developers were trying to make complex programming easier by increasing the abstraction levels.
  • Various Programming Languages emerged supporting various paradigms.  Object Oriented, Functional, Procedural, Logical etc, came in being during this period.
  • C, Pascal, SQL all came into being during this period.
  • A major shift was under way to make programs easier to develop and maintain through new language enhancements and programming paradigms.
  • Towards the 1980’s emphasis turned towards paradigms and language maturation.
  • C++ which came in ’79 was originally called C with Classes, brought the advantages of Object Oriented Programming to a language strong in systems programming.

Productivity

  • In the 90’s the developer productivity was the driver of the languages evolution.
  • Even more complex programming was abstracted out.
  • Runtime Environments tried to remove the burden of memory allocation and de-allocation etc from the programmer, and there by concentrating him only on the essence of programming for the problems.
  • Integrated Development Environments started to come in.  And then, the World Wide Web came in.
  • Java came to the world as a language which could execute on top of any environment and a lot of operating systems.
  • Simplicity, Portability, removing unnecessary programming overheads (memory allocation, management etc.) became the force behind the evolution.
  • Then came support for programs written in multiple languages.
  • JVM and .nET Common Language Runtime provided platform support for multiple languages.
  • This platform support for many languages eliminated the support for many a programming language.
  • Then came in the frameworks, which tried to abstract out even more of what was complex and repeatedly used programs.
  • But these in themselves introduced the same problem which they tried to solve.
  • To remove one complexity, it introduced another one.  Coz sometimes learning a framework was as daunting as learning a new language itself.

The Future

  • The future is interesting.
  • Just because we have seen the final programming languages of today’s era, it does not mean that programming languages are through evolving.
  • There are even more complex programming puzzles to solve like NLP, fuzzy, Expert Systems etc.
  • The Evolutionary process is just beginning & will have many more surprises for us.

Pros

Loops & Conditions :
We have all the beautiful pieces to solve the puzzle, like if….Then…Else,  for,  while,  do…while,  GOTO
Design Patterns :
What do you want to do :
Can do anything and everything.

Cons

Can be Complex:
Sometimes going thru the code, you may find it like puzzle in itself.  could be confusing.  you can get easily lost.
Tower of Babel:
There are toooo many languages today.  Each have their own distinctions, but still, can’t there be just one language.

Dream,

For, In a way,

We (programmers) are God.

Linux, Office

Installing Ingres….

I was little unwell last week and before, and not being able to do the work i was doing, i was moving around the office, When Balaji asked me to help him to install linux.  Being Me, i asked why you want to install Linux?  He said, he had to test if the enterprise version of our product will work on ingres database.  My First thoughts were (seriously) WTF is ingress(later i found that it was ingres and not ingress)?  And that is how i ended up doing it for him, and that is why i am here writing this.  I wanted to write it here, only after completing the entire exercise.  or atleast after everything was completed.  (Connecting from a machine other than the server(localhost) is not yet working).  But anyway connectivity issues and installation are different.  So without bragging and dragging, let us proceed with how to install Ingres in Linux.
For this exercise i used a RHEL5 machine setup on VirtualBox.  You need to have UI.  I am listing those steps which i had recieved as instructions when i had called up technical support of Ingres.
  1. create a user called ingres.
  2. create a group called ingres.
  3. add the user ingres to the group ingres.
  4. login as root.  yes you read that right, no logging in as normal user and changing into root by issuing switch user command.  the guy gave up on me when i told i did a su to become root and did the install, when we faced the connectivity issue.  So better login as root, if you might be calling the tech support any time.
  5. create two directories ingres_src and ingresII under /opt. (it ain’t case sensitive, you can name it anyway, but the guy said it is his way that he always keeps the names in small case, unless and until mentioned not to)
  6. move the installation gzip file and unzip it into ingres_src folder.
  7. change into that directory and execute ./ingres_install
  8. In the First Screen choose your Configuration Type.  I choose Transactional System.
  9. In the next Screen choose the Installation Mode as ‘Advanced‘.
  10. Instance Name should be IngresII (two capital I)
  11. In the Component Select Screen select Custom.  Uncheck Ingres Replicator and Ingres Star.
  12. Set the Installation Location to the folder which we created next /opt/ingresII.  Leave the rest as such
  13. Set the transaction log file size to half the size of the total RAM available to the system.  If you would be running any other applications in the machine, then calculate the memory with what would be left after satisfying the other applications memory requirements.
  14. Set your locale settings.  Do not change the Character Set.
  15. Do not enable Strict SQL Compliance
  16. Choose the Date Type as ANSI Date
  17. Set the Instance Owner user as ingres and group as ingres
  18. UnCheck Start Ingres with computer, and Check create a demo database.
  19. Well that’s it, Ingres should successfully install and start.
To make sure that the Installer started the server, login as ingres user and issue the command ingstatus.  If you get error saying command not found or something like that, you better logout and then login.  If it still does not work, go the home folder of the user ingres and look for file named .ingIIbash and .ingIItcsh, copy the contents of .ingIIbash to .bash_profile of ingres user and then logout and login again. now it will work.  The output will be something as follows
Now if it hasn’t started issue the command ingstart and wait till the control returns to you.  now in terminal type
sql demodb
and press enter, you will be taken into the screen where you can execute the queries. to come out of it type \q and press enter.  to stop the server use the command ingstop.
Linux

Windows as a Guest on a Linux (ubuntu) host

After having been through all those days of triple boot and quadruple boot systems, i am coming back to a single boot machine now.  This time with my work laptop with Ubuntu as the main and the only OS.  Since i knew i could never complete that dream of running everything from linux in my work place in the near future, i decided to go with the only other choice to achieve it with immediate effect.

So what are my work place requirements. i need to have a firefox with silverlight.  That is the only thing for which i need Windows, ok, may be occasionaly for something else like, i don’t know…. I can’t think of any thing else.  and though even that can be circumnavigated with Wine, the video chat on gmail is another thing which will become a road block.

So to solve these, i decided to install Windows on Virtual Box with Ubuntu as Host.  This i thought will solve the problem.  But, yeah it did solve the problem of firefox with silverlight.  But Virtual Box is not allowing to enable to use of webcam to the guest OS.  It has to be a small problem somewhere i think.  Will look into it.  And to solve the problem of not having a anti-virus, I am allowing internet only to firefox, and have removed the access to the rest of the system.  I did it with Squid on the host.  It feels good, to be close to achieving the original thought.  But i can live with a little Evil i guess.

For Virtual Box installation i followed the instructions in this page under the title Debian Based Distributions.  Just in case if you are not able to open it.

add the following line in your repositories list in Synaptic Package Manager (settings -> repositories -> other software)

deb http://download.virtualbox.org/virtualbox/debian lucid non-free

add the key to the Synaptic Package Manager (settings -> repositories -> authentication)

then say

sudo apt-get update
sudo apt-get install virtualbox-3.2

That should do it.  I mean that is all i did. and i had to go thru this to understand how to access shared folders form host to guest and why the display resolution went to 800×600 when i made the shared folder access to work.  It was frustrating, the display always going to 800×600 from 1024×768  finally figured out, that it was the Virtual Box Display Driver which was causing it.  Then i just uninstalled just the display driver, and override it with the default driver whatever windows took itself.  And that is it.  Things are working fine for now.  Just the small thorn of WebCam access from host to guest.  Will soon try to find a way out.