Sunday, September 8, 2019

Basics to Understand Modern Infrastructure


When discussing modern infrastructure or cloud infrastructure lots of terms are tossed around like virtualization, container, docker, etc. While it is not essential for everyone to understand these in detail, a basic/fundamental understanding/view is becoming essential for those who are  in the field of software development.

In fact, even a lot of people who may already use one or more of these, may still have some "vagueness" when it comes to explaining them to others. So, I wanted to capture some quick points which helped me have a high level  understanding of these. These may not be 100% technically correct, but should help the aforementioned purpose.

Virtualization / Virtual Machine (VM)

This once is fairly straight forward and most of us have a clear view on this. We have a physical machine with certain resources. Slicing this into multiple machines virtually is called virtualization. The OS running on these machines will see them as a physical machine. By the way, the physical machine is referred as the metal or bare metal servers. Sometimes they are also referred as host machines. The slices are called virtual machines or VMs.

Vagrant

When slicing the physical machine into multiple into virtual ones, we need to provide the definition, including its software configuration, for each of these machines. As multiple virtualization tools came up, there was a need to create this definition that was not tied to a specific tool. Vagrant is a tool that fills that need. Create the definition as a vagrant definition and use that definition to run the VM and the software setup on any virtualization tool.
Container and Docker

A container is actually a combination features of Linux kernel which provide an isolation to a group of processes. For this reason they are also called as OS level virtualization. While the features required for this like cgroups and namespace isolation are available at the OS level, container tools offer them as a bundle, hiding the details. See LXC

Docker is a tool that helps to define such containers with a specific run time like OS etc. and then run them by leveraging the container features of Linux. See here for more details and technically correct information.

Kubernetes
Kubernetes (you would see this also referred as k8s) is a container orchestration platform. Instead of dealing with individual containers, Kubernetes deals with what's called a Pod, that represent the group of resources for an application. It then manages them by scaling the number of instances, choosing the node to run them, monitor, restart etc. See here for more information.

Thursday, August 29, 2019

Starter Toolkit for Programming Beginners

One of the things that is going on in my mind for a while was creating a quick guide on the toolkit for programming beginners. When I say toolkit, I don't mean the tools like IDE etc., but rather the skills and nuances that are mostly learnt on the job, assuming that we get to have a good mentor. It is not uncommon for even programmers with enough experience in the industry not getting a chance to acquire these skills.

Some of the Gen Y/Z kids may have got some insight into some of these aspects, thanks to the active opensource communities. But, still collating these in one place so that they get a view or an introduction to most of these by the time to get into full-time programming.

I am planning to expand on each of these topics, as the time permits and would be linking to the blog article as I write them.

Here is the inventory of items that are part of the tool kit, with predominant focus on model web applications and mobile apps:

  • Understanding of the networking involved - the client browser to multiple servers where different pieces even seeming simple websites exist, the overwhelming TLAs invented to scare the new comers, etc.
  • Understanding what happens in the server and what happens in the client (the web and mobile)
  • Security - What are the basics to know and modern security concepts
  • Source control or version control - How to protect your work from getting lost
  • Refactoring - The art of routing maintenance of the code you write 
  • Debugging anything
  • Scripting tools a.k.a. Automating boring work
  • *nix Basics - Programmers paradise

Monday, October 30, 2017

Refactoring Code

I believe refactoring is a very important trait for a programmer. Refactoring helps in reducing the repetition of code and also helps to make the code generic. Like everything else in life, code too accumulates dust if not maintained constantly. Constant refactoring helps in keeping the code clean. Learning to refactor early on in programming career helps in having a great career. In fact it is something we don't even need to learn as, as human beings we are experts at identifying patterns and  crux of refactoring is pattern matching.

Recently I was reminded of the age old programming puzzle by a friend and wanted to use that to capture my thoughts on refactoring.

The code is to generate something like this:



The initial code we would typically write would be specific to this particular output . (See the initial iteration in JSBin)




The first thing that strikes from this code is the repetition of the code block within the two for loops. The first refactor will be to extract this into a function. 

Anytime we see code being repeated, it immediately calls for refactoring. It is well known that lesser the code lesser will be the defects. 

The refactored code could look like this: (1st round of refactor)

This is still specific to the particular output. It could further be refactored to be generic, suitable for output of similar pattern, say a grid of 15x15. (2nd Round - Generalization)


Still it provides the same output as the first code, but, in a much generic way, with less repetition. 

The key rules of refactoring:

#1: If there is repetition of code, it is a no brainer for refactoring

#2: If the code could be applicable for similar problems, refactor it to be generic. Just be aware of the ROTI (Return on time invested)

#3: Don't make the code clever in the name of refactoring. The code should be clear to understand. Not clever. This code could be refactored to use a single loop, but, at the cost of clarity.   See line 24 in the code below. (This is fairly straight forward than the clever codes out in the wild. But, you get the drift.)


(Have a look at http://jsbin.com/bemitew/edit?html,js,output which has this 1 loop implementation)

Happy refactoring!




Monday, March 13, 2017

Installing QBasic on Mac a.k.a Revisiting the school days

My daughter recently needed QBasic for her school work. I set out to install the exact same program in my Mac as online emulators did not do a great job without confusing her.

The ingredients:

  • Oracle VirtualBox
  • FreeDos
  • olddos.exe
Recipe:

  • Installed Oracle VirtualBox
  • Installed FreeDos using (http://wiki.freedos.org/wiki/index.php/VirtualBox), along with networking
  • Created a ISO image with olddos.exe ( https://web.archive.org/web/20070316205657/http://download.microsoft.com/download/win95upg/tool_s/1.0/w95/en-us/olddos.exe )
    • Used DiskUtility and created a new image (Be sure to choose image format as CD/DVD Master)
    • Copies the olddos.exe to the image
    • Used the following command from terminal to convert the CDR to ISO
hdiutil makehybrid -iso -joliet -o DosUtil.iso DosUtil.cdr

  • Mounted the ISO in VirtualBox as  a drive
  • Ran the self extracting archive olddos.exe
Now QBasic is all hot and steaming and ready to be served.

Monday, July 8, 2013

Dispatcher Servlet to turn off handler stopping at semicolon


 import org.apache.commons.lang.StringUtils;
import org.springframework.beans.factory.BeanFactoryUtils;
import org.springframework.web.servlet.DispatcherServlet;
import org.springframework.web.servlet.handler.AbstractHandlerMapping;
import org.springframework.context.ApplicationContext; 

public class MyDispatcherServlet extends DispatcherServlet {
    @Override
    protected void initStrategies(ApplicationContext context) {
        super.initStrategies(context);
        for (AbstractHandlerMapping handlerMapping : BeanFactoryUtils.beansOfTypeIncludingAncestors(
                context, AbstractHandlerMapping.class, true, false).values()) {

            handlerMapping.setRemoveSemicolonContent(false);
        }
    }

}

Wednesday, April 17, 2013

My Ordeal with Samsung S2 JB Upgrade (Has happy ending)

I have been waiting for the JB upgrade for my GT-I9100G for quite sometime. It is phone bought in India and the update has been rolling out gradually in the past weeks for S2 s in India. I was clicking on the "Check for updates" button everyday, in the hopes of getting the update even though I had enabled automatic update check.

Last Friday, I was very happy to see my screen popup with the information that the update has been found and downloaded and prompted me to install or later. Usually I make sure that the battery power is almost full and then initiate the update. But, this time my excitement overtook my caution and I clicked on "Install". I immediately plugged in my phone into the power socket and waiting for the update to complete. It seemed to be running fine, but suddenly after the automatic reboot, it had got stuck in the boot screen. I waited for quite sometime before I realized that my phone has been "soft-bricked".

I had also not backed up some of my data in my excitement and my ordeal began. My tasks were to:
1. Get the data from the internal SD out
2. Update to JB

I tried entering into the Android recovery mode  (Volume up + Power + Home) hoping it to be helpful. It did not help much with respect to getting the update work. Then after searching the net for a while I could get the required tools for data recovery and update. All the information is out there, but, I felt they were are not in a single place.

The first step was to get to a better recovery mode of ClockWorkMod. To install this, I did the following steps:

1. Odin307.zip  - http://forum.xda-developers.com/showthread.php?t=1738841
2. Installed CWM for recovery mode -
a. GT-I9100G_Blazing_Kernel_v3_CWM5.Tar from http://www.androidfilehost.com/?fid=9390288116658474142
       b. Used Odin to install this Kernel. The instructions are commonly available:


  1. Turn off you phone and enter Download or Odin mode: press and hold Volume Down + Home + Power buttons together for a few seconds.
  2. You will  see a screen asking you to press the Volume Up button. Press it and you will enter the download mode.
  3. Launch Odin
  4. Connect your phone to the computer via USB cable and wait till Odin detects your device. A successful connection is indicated by the ID:COM port turning blue and “Added!!” text at the message box below.
  5. Make sure that only the “Auto Reboot” and “F Reset Time” options are checked on Odin. 
3. Launched the CWM recovery mode by using volume up+ power + home buttons. 
4. Used CWM backup to SD card to create back of phone's system storage. - This back up contains phone logs, SMS and contacts.
5. Use the CWM mounts menu item to mount internal storage (sometimes referred as internal SD) of the phone. It was called as emmc for me. (Some people have mentioned that their internal storage was called as sdcard and external as emmc)
6. Connect the USB cable and use "adb pull" (Android developer tools) get the files in emmc 
7. The contacts will be available in data\data\com.android.providers.contacts\databases directory of data.ext4 zip file in the CWM back up created in step 4. Use the script from https://github.com/stachre/dump-contacts2db to convert contacts2.db to VCF which could be imported into Android. I used a Ubuntu running on a Sun VirtualBox to execute the script. Use the following command in Ubuntu to install required dependencies for the script. 
sudo apt-get install sqlite3 libsqlite3-dev.
8. SMS are available in data\data\com.android.providers.telephony\databases\mmssms.db. Used yaffs-mmssmsdb-calls-extractor.zip to get the SMS from backup to the XML  format used by the "SMS back & restore" android application, which could be imported. The application can directly work on mmssms.db. 
9. Now the back ups are complete. The next is to install the JB. The official Samsung JB build is available from http://www.sammobile.com/firmwares/1/?model=GT-I9100G&pcode=INU#firmware for India model of GT-I9100G. 
10. Then use Odin to write the firmware
11. Restore contacts and SMS. Use import menu in contacts to restore VCF. Use SMS back &  restore for importing back the SMS.  



At last the S2 was back in business.

Thursday, April 11, 2013

File download page using Spring


In web.xml, map directory listing URL to spring dispatcher:


 <servlet-mapping>
  <servlet-name>spring</servlet-name>
  <url-pattern>*.html</url-pattern>
 </servlet-mapping>

 <servlet-mapping>
  <servlet-name>spring</servlet-name>
  <url-pattern>/download/*</url-pattern>
 </servlet-mapping>

The first one is your existing dispatcher and the second one for download.

In spring security configuration set up appropriate configuration:
<intercept-url pattern="/download/**" access="permitAll" />

In Spring servlet config xml set this up to make path configurable:

 <bean id="appPropertiesBean" class="org.springframework.beans.factory.config.PropertiesFactoryBean">
  <property name="singleton" value="true" />
  <property name="properties">
   <props>
    <prop key="downloadPath">C:/</prop>
   </props>
  </property>
 </bean>

In your controller for the path property:

    @Value("#{appPropertiesBean.downloadPath}")
    private File downloadPath;

In controller add methods to generate directory listing:

   @RequestMapping(value = "/files/**", method = RequestMethod.GET)

    public ModelAndView archiveDirectoryListing(HttpServletRequest request, HttpServletResponse response) {
        try {
            //String restOfTheUrl=(String)request.getAttribute( HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE );
            //The above line seems to cause issues with filenames 
            // containing semicolon. Use the below line instead
            String restOfTheUrl=request.getPathInfo();
            restOfTheUrl=restOfTheUrl.substring("/files".length());
            File physicalFileRequested=new File(downloadPath,restOfTheUrl);
            System.out.println("Accessing .." + physicalFileRequested);
            if (!physicalFileRequested.getCanonicalPath().startsWith(archivePath.getCanonicalPath())) { //Requester is trying to go above the archive directory
         response.setStatus(HttpServletResponse.SC_FORBIDDEN);
         return;
            }
            if (!physicalFileRequested.exists()) {
         response.setStatus(HttpServletResponse.SC_FORBIDDEN);
         return;       
            }         
            System.out.println("Access allowed " + physicalFileRequested);
            System.out.println("Access allowed " + physicalFileRequested.isFile());
            System.out.println("Access allowed " + physicalFileRequested.isDirectory());
            System.out.println("Rest of url : " + restOfTheUrl);
            if (physicalFileRequested.isFile()) {
         System.out.println("Accessing file:" + physicalFileRequested);
         streamFile(physicalFileRequested,response);
            } else if (physicalFileRequested.isDirectory()){
         if (! restOfTheUrl.endsWith("/")) {
             response.sendRedirect(request.getContextPath() + "/download/files" + restOfTheUrl + "/");
         }
         System.out.println("Accessing dir:" + physicalFileRequested);
         generateDirectoryListing(physicalFileRequested,response);
            }
        } catch (Exception e) {
            e.printStackTrace();
            throw new RuntimeException("Error accessing requested file/directory.");
        }
    }

    private void streamFile(File physicalFileRequested, HttpServletResponse response) {
 try {
     response.setContentType("application/pdf");
     // get your file as InputStream
     InputStream is = new FileInputStream(physicalFileRequested);
     // copy it to response's OutputStream
     IOUtils.copy(is, response.getOutputStream());
     response.flushBuffer();
 } catch (IOException ex) {
     throw new RuntimeException("Error accessing requested file/directory.");
 }
    }

    private void generateDirectoryListing(File physicalFileRequested, HttpServletResponse response) {
 try {
     StringBuilder sb=new StringBuilder();
     sb.append("<html>");
     sb.append("<body>");
     sb.append("<a href='..'>..</a><br>");

     File [] childFiles=physicalFileRequested.listFiles();
     for (File childFile:childFiles) {
  String relativePath = childFile.getName();
   sb.append("<a href='"+ URLEncoder.encode(relativePath,"UTF-8")  + "'>" +relativePath+"</a><br>");
     } 
     sb.append("</body>");
     sb.append("</html>");
    response.getWriter().print(sb.toString());
 } catch (Exception e) {
     throw new RuntimeException("Error accessing requested file/directory.");
 }
    }



Download URL will be like:

http://localhost:9090/context/download/files/

Alternatively, the directory listing HTML could be generated using a view, by using the following alternate implementation of the method (Note: wherever null is returned for ModelAndView, Spring assumes that response has already been handled and no view redirection needs to happen):


   @RequestMapping(value = "/files/**", method = RequestMethod.GET)
    public ModelAndView archiveDirectoryListing(HttpServletRequest request, HttpServletResponse response) {
        try {
            String restOfTheUrl=(String)request.getAttribute( HandlerMapping.PATH_WITHIN_HANDLER_MAPPING_ATTRIBUTE );
            restOfTheUrl=restOfTheUrl.substring("/files".length());
            File physicalFileRequested=new File(archivePath,restOfTheUrl);
            System.out.println("Accessing .." + physicalFileRequested);
            if (!physicalFileRequested.getCanonicalPath().startsWith(archivePath.getCanonicalPath())) { //Requester is trying to go above the archive directory
         response.setStatus(HttpServletResponse.SC_FORBIDDEN);
         return null;
            }
            if (!physicalFileRequested.exists()) {
         response.setStatus(HttpServletResponse.SC_NOT_FOUND);
         return null;         
            }           
            if (physicalFileRequested.isFile()) {
         streamFile(physicalFileRequested,response);
         return null;
            } else if (physicalFileRequested.isDirectory()){
         if (! restOfTheUrl.endsWith("/")) {
             response.sendRedirect(request.getContextPath() + "/archive/files" + restOfTheUrl + "/");
         }
         System.out.println("Accessing dir:" + physicalFileRequested);
         ModelAndView modelAndView=new ModelAndView("direcoryListing");
         modelAndView.addObject("fileList", physicalFileRequested.listFiles());
         return modelAndView;
            }
        } catch (Exception e) {
            e.printStackTrace();
            throw new RuntimeException("Error accessing requested file/directory.");
        }
 response.setStatus(HttpServletResponse.SC_NOT_FOUND);
 return null;         
    }


And a freemarker view, direcoryListing.ftl:

<html>
<body>
<h1>Welcome to the download page</h1>
<a href='..'>..</a><br>
<#list fileList as file>
<a href='${file.name}'> ${file.name}</a><br>
</#list>
</body>
</html>

Configure freemarker in application context:

<!-- freemarker config -->
<bean id="freemarkerConfig"
class="org.springframework.web.servlet.view.freemarker.FreeMarkerConfigurer">
<property name="templateLoaderPath" value="/WEB-INF/pages/" />
</bean>           
<!-- View resolvers can also be configured with ResourceBundles or XML files. 
If you need different view resolving based on Locale, you have to use the 
resource bundle resolver. -->
<bean id="viewResolver"
class="org.springframework.web.servlet.view.freemarker.FreeMarkerViewResolver">
<property name="cache" value="true" />
<property name="prefix" value="" />
<property name="suffix" value=".ftl" />
</bean>