programming

getting cidr from the ipaddress / subnet

I have been working on the hiera data generation / validation process. for generation of certain detail i needed both the ip address in the usual 1.1.1.1 way and cidr way. I had initially asked for both as input, when the user’s came back saying what if i give the subnet information along with the ip address like 1.1.1.1/9 and you can calculate the CIDR information. Sounds right.

I had initially asked for that from the user because i did not know how to calculate thing. But since the user has asked it to be that way, i started to look into how to arrive that value. I had the first clue with me, the user had told to look in to XOR based calculation. I started with a generic search to first understand what it and how to arrive it.

May be my Search fu is not very good or whatever, my search always lead to getting ip ranges / calculating the subnet mask with CIDR, and not how to arrive at the CIDR.

There were a few calculators which gave what i wanted, but did not explain how to arrive at it. but playing around with them and seeing the outputs, i could sense a sort of pattern emerging, then thought i have seen something like this before. Then i remembered that there was a third party package which some other application was using at work. i searched for it and lo and behold python3 has that function built in. not sure if python2 had it or not.

Python3 has this module ipaddress which can be used for ip address related things. so for my use case all i had to do is call the appropriate function and give the ipaddress/subnet information and get the output. I had to set strict argument to False. so that i would take ip address also as part of input instead of proper cidr.

>>> import ipaddress
>>> network = ipaddress.IPv4Network("10.25.123.20/20", strict=False)
>>> str(network)
'10.25.112.0/20'
>>> 

Office, programming

Testing hiera data

I have been looking around for ways to validate puppet hiera data files.

my requirements were

  • check if the yaml is well formed.
  • check if all required settings are provided in the yamls
  • check if the required settings values are of expected format
  • inform the how a value is being considered when the setting is present in multiple files in hierarchy.

The basic idea that i had arrived at was to use json schema based validation. Basically create schemas for the various files in the Hiera Hierarchy, and validate each file against its own schema.

I already had a toy project which does that, which i made sometime last year. But, thought would research a little bit and do things a little properly.

I was curious to see if this problem hasn’t been faced by others or not. Searching around the internet found that puppet has a command which sort of does this. it’s not exactly a validation of the data, but what would hiera return when the setting is being requested.

puppet lookup command

That still wouldn’t help in our case. we can see what setting would be applied, but if the engineer has filled it in the correct format or not, if this is a boolean or string, those cannot be validated. also we can lookup only one setting at a time. But it helps in seeing how puppet arrives at the value with the helpful “explain” option.

on further searching i found that the exact same idea has been thought off and they also referred to tools which would help to achieve that.

https://logicminds.github.io/blog/2016-01-16-testing-hiera-data/

The post describes about puppet-retrospec and kwalify to achieve that. but those didn’t work out much, seems they are for older version of puppet.

for ensuring if the yaml is right or not, thought there should be something like pylint or jslint, and i was right, a simple search pointed me to something called yamllint. Which also had a python interface with which it can be included into the our scripts. a simple example of it would be like

from yamllint import linter
from yamllint.config import YamlLintConfig

conf = YamlLintConfig('extends: default')

filepath = "./test.yaml"
with open(filepath) as yamlfile:
    yaml = yamlfile.read()
problems = linter.run(yaml, conf, filepath)
for problem in problems:
    print(f"{problem.line}:{problem.column} \t {problem.level} \t {problem.desc} ({problem.rule})")

which produces

$ python testyamllint.py 
1:1      warning         missing document start "---" (document-start)
3:1      error   duplication of key "a" in mapping (key-duplicates)
3:7      error   no new line character at the end of file (new-line-at-end-of-file)

for a sample yaml

a: "b"
b: "c"
a: "d"

let’s see how it goes.

programming

Sorting Coursera Course List by duration

With the lockdown going on, was too bored. Work from Home, playing with kids and playing Injustice, have sort of become a routine. The first weeks on lockdown I spent the spare time watching Arrow on Airtel Xstream. which suddenly disappeared from the platform. didn’t know what else to do, Coursera announced a free courses saw the cloud computing 101 and thought would refresh my memory on the subject. It was 3 weeks long. With byte sized videos and mostly theory i sort of finished it in 1.5 – 2 weeks. So, thought would look into some other short duration courses.

Coursera page has a categories list and once you select a category from the filters you can select duration and get a list of courses within the selected duration. But then you cannot sort by duration.

I first looked to see if Coursera offered any APIs. They do provide but you have to join their affiliate program. I just want to sort by duration. So this will not be helpful for us.

So I decided to see what happens behind the scenes when I set a filter and then use that to get the data and sort it separately.

So what happens behind the scenes

When checking with dev tools, i observed the following. When we select a filter and click on Apply Filters, a POST request is sent to this URL

https://www.coursera.org/graphqlBatch?opname=catalogResultQuery

that doesn’t seem right, right? the filter information is not there. That information is being sent as post data. Checking the payload, there was this huge JSON data which was being sent. It had all the information, a huge query which coursera backend uses to fetch the required data i guess. but the thing which we are interested in is the variables object

{
  "limit":30,
  "facets":[
    "skillNameMultiTag",
    "jobTitleMultiTag",
    "difficultyLevelTag",
    "languages",
    "productDurationEnum:1-4 Weeks",
    "entityTypeTag",
    "partnerMultiTag",
    "categoryMultiTag:information-technology",
    "subcategoryMultiTag"
  ],
  "sortField":"",
  "start":"0",
  "skip":false
}

the information that we would have to manipulate are limit, start and the value “categoryMultiTag:information-technology” inside the facets list. the limit and start are used to control pagination, and the number of courses to return in a single request. and categoryMultiTag is the course category that we are interested in. Making a request to this URL we get a JSON response, which has the courses information and the pagination.

The pagination details are found under [“data”][“CatalogResultsV2Resource”][“browseV2”][“paging”] which has total number of courses and the next request start value.

The courses list is under [“data”][“CatalogResultsV2Resource”][“browseV2”][“elements”][0][“courses”][“elements”].

Now that we have the required information of where to get the information, what to send and what to expect in response, I wrote a script to recursively get all courses of a category and duration and dump it to csv.

you can find the entire script here https://gist.github.com/anabarasan/e6b5b6842e97592ec1eaffbc30ce703e

Office

GlassFish v2.1.1 & Custom Error Messages

There are times when you might want to enable custom exotic jazzy error messages instead of the boring 404 or 500 messages. It can be due to various reasons. Security, to avoid scarring the users, etc.,

You can enable custom error messages either server wide (all applications in a server) or only to a specific application in the server.

To enable it across all applications :

This is specific to Glassfish v2.1.1, for the other versions it may vary.

Method 1:  editing the domain.xml manually.

  1. goto glassfish\domains\{domain directory}\config\domain.xml
  2. copy the custom error message html files here.
  3. Open the domain.xml file located in the same folder.
  4. Find the tag “virtual-server”, which has an attribute “id” with the value “server”
  5. Add a sub element to that virtual-server element like this.

<property name=”send-error_1″ value=”code=404 path=404.html”/>

the name attribute value has to be of the format send-error_n where n is a number. for different errors increase the value of n and change the value of value attribute.    The value has two parts code and path.    code is the three digit HTTP response code, eg: 400, 404, 500, 502 etc..    the value of path is the path to the custom error html page. this path is relative to the position of this domain.xml.

Method 2:  Via Admin Console.

  1. login to Administration Console.
  2. In the tasks menu on the left, goto Configuration => HTTP Service => Virtual Servers => server
  3. In the right pane, scroll down to the bottom, and click on the Add Property button.
  4. under name enter “send-error_1”, and under value enter “code=404 path=404.html”
  5. similarly for every error add a property and enter the values, for explanation on values for name and value refer to Method 1.

To enable it to a specific deployed application :

Custom error messages can be enabled for a particular deployed application, by modifying the deployment descriptor file. In web.xml file, under the web-app element add the following,

<error-page>
          <error-code>404</error-code>
          <location>/404.html</location>
</error-page>

You can add multiple error-page element for each error. the location, is the path to the custom error page.   The path is relative to the docroot of the application.

Reference:
http://docs.oracle.com/cd/E13222_01/wls/docs81/webapp/web_xml.html
http://docs.oracle.com/cd/E19879-01/821-0183/abhaq/index.html
http://docs.oracle.com/cd/E19879-01/821-0183/abhfg/index.html

Office

Belling the Cat (or) How i made OrangeScape to tango with Tomcat

A month or two ago, a few colleagues of mine, wanted to run OrangeScape in Tomcat for their local testing,   just because they didn’t like jboss and run it.  I wanted them to run through the internet and do it by themselves, so i can chance upon and see if anyone is up there to pull into my team when the time comes.  But they got busy with other things, and forgot about running OrangeScape in Tomcat, and went with jboss.  Now, i am fed up of waiting.  So here i am documenting my findings on how to make OrangeScape tango with TomCat.

I have tested with  Tomcat 6.0.35 and 7.0.23 with JDK 1.6.29.   Running OrangeScape Application in Tomcat is really easier than what i had expected.  really.

Getting the Application War files

You can download the war of your application from the studio.  download the supporting war files (os-commons, runtime, static) from the corresponding build release notes from the community.  The Current release number being 98.

Making changes to make it work in Tomcat

You will have to make changes to 2 files for the OrangeScape Application to work in Tomcat.

    • app.war\WEB-INF\applicationContext.xml
      • search and find the line
        <prop key=”hibernate.connection.datastore”>jdbc/{application-id}</prop>
      • change it to look as below
        <prop key=”hibernate.connection.datastore”>java:comp/env/jdbc/{application-id}</prop>
    • app.war\META-INF\context.xml
      • this file does not exist in the war and you will have to create it. the content should be as follows.
        <?xml version=”1.0″ encoding=”UTF-8″ ?>
        <Context>
                <Resource
                        name=”jdbc/{application-id}”
                        auth=”Container” type=”javax.sql.DataSource”
                        username=”{db-user- id}” password=”{db-user-password}”
                        driverClassName=”com.mysql.jdbc.Driver”
                        url=”jdbc:mysql://localhost:3306/{db-name}”
                        maxActive=”20″
                        maxIdle=”5″
                />
      </Context>

That’s it.  Now, just deploy it to tomcat, and access the url http://localhost:8080/{application-id}/1/signin to log in and start using your application.

Note:

  • The first time, you start the server after making the deployment,  (immediately after deployment, if you use the manager to deploy), you can see a HTTP 500  error will be thrown on the console.  That is because the application is not yet setup but the sla is checking if any jobs are there to execute.
  • as far as i have checked things work.  Webservices & reports were throwing errors. But it could be due to some mistake of mine, since i was in a hurry, to go to sleep.UPDATE:  Everything works fine, as i had noted, it was because of one of my mistakes.

Reference:

Office

User Management via SQL for OrangeScape Applications

User Management, is one of the new functionalities available in OrangeScape Applications created after the late september release. For those applications which were created before that release, during migration, the required fields would have been created. But, the developers will have to make the additional changes like, process, forms by going thru the User Management available in the newer applications.

This post is not about explaining the UserManagement part or about what changes have to be made for UserManagement to work in older applications. This post is about creating users and managing user roles via the backend, directly in the database.

First a small understanding about the models of User Management. UserManagement is made of the following models: User, UserChanges, RoleSubscription, AppRole


I know the image is not so good. The diamonds are sightly wrong. Hope, atleast you will be able to understand the one and many sides. Ok.

So there is the AppRole model, which has all the roles which you have defined in the application. You don’t have to insert anything there, the application reads your process and creates the required records for the roles defined there. Next there is the User model, where all the user records are stored. Then There is the Role Subscription model, where you define the roles to which the user is assigned / subscribed to. The UserChanges model has the change history of the user’s details.

When you create a user a record is created in UserChanges model, and all required details are entered. Once you save the record, the actual User record is created. Once this is done, you can assign roles to the User, by inserting records to the RoleSubscription Model. To know how to do these things from the application, please refer the corresponding section in http://learn.orangescape.com.

If you are wondering why the initial User details are being captured in UserDetails,. Once a record is created in User model for a User, the whole process of user creation is complete. When you do a create new in Orangescape Application, that record is immediately saved in the backend, and then only it is given for you to edit. Which means, when doing a create new in User Model, a new, empty User record will be created. which is BAD. I you want to know how this data entered in UserChanges syncs up with User and how all changes done for the User in UserChanges are listed from User, read the post here.

Ok. so now, we know a little of how User Management Works, (any questions / doubts, for the better good of other people, please ask them in the community forums). There is a beautiful & easy UI provided in the application. Make sure you do check it out. But, as everyone knows, it is a pain to enter 60 or 70 user records. Well, then you have the CSV import option available, which even validates if your data is right or not. But, for some, for some reason, they need to insert data into the database directly. (In the case of integration). Lets proceed with inserting data directly back in to the backend.

NOTE:

This is for those versions where you use a database for backend. On Premise or GAE SQL. not for GAE noSQL (BigTABLE).

download this SQL. It has the sample queries, and the minimum required Columns and the data required for those columns. use those queries as templates, and insert the data as required in the backend.   Since updates would never make sense, with Cache in place, Modification are to be done via the application.

Office, programming, What the...

Birthday Alerts app [version 2]

When i wrote last time how i made the birthday alerts app , i wrote, how, because of a property of SLA, the data had to be entered at that time of the day when you want to receive on the desired day.  (I know i am being confusing because i am trying to say it all in one line, read the previous post and you will understand).  Well now you don’t have to,  and have made the logic behind the app mucho simpler.  Well, the process flow part is just one activity, apart from start and end, unlike the previous time, when it was like, 7 or 8 activities.  It is still 7 fields, and all logics have been moved to actions.

The activity Reminders has  a SLA set on it.  The time duration is obtained from the field RemindIn.  The two actions in this are, one to calculate the time, when to remind and the other one, to send the reminder and reset it for the next round.  The changed datamodel is as follows.

The calculations done to get the RemindIn time duration is as follows.

  • DayNextYear = (DATE(YEAR(NOW())+1,MONTH(Birthday),DAY(Birthday)))
  • DayThisYear = (DATE(YEAR(NOW()),MONTH(Birthday),DAY(Birthday)))
  • RemindIn = IF(Birthday>NOW(),(Birthday-TODAY()-1)*24+HOUR((TODAY()+1)-NOW()),IF(DayThisYear>NOW(),(DayThisYear-TODAY()-1)*24+HOUR((TODAY()+1)-NOW()),(DayNextYear-TODAY()-1)*24+HOUR((TODAY()+1)-NOW())))
Well that’s it.  Think anything is amiss, or some other thing, that should be there, lemme know.
About this app:
This is a simple app, which takes in the date on which you have to be reminded, and sends you a mail about what you want to be reminded.   It will automatically remind you every year [since, you see, birthday’s / Anniversaries come only yearly].