05 July 2019

How do I deliver ALAMI's Prod

Back to two months ago when ALAMI still under registration process by Financial Service Authorities in Indonesia (OJK), all of team has been fight to fit our platform application with OJK regulations. After the final stage, we're still doesn't have a devops for creating an infrastructure of our application to be in production (live). So the CTO ask me to learn quickly about devops and creating minimal system to be used for our application in production. And then the journey has began.
Actually we have our old server but the Management doesn't happy with the provider, so we are searching new provider. How lucky we are, since our office are located at coworking space, there're many startups and one of them have a cloud service named ZettaGrid. After some meetings, the Management team interested to use their services. I got the free-trial Virtual Data Center (VDC) to exploring all of features that they have. But time is running low before we're going live, so I need to accelerate my exploration.
I has designing all of the entire infrastructure that would be used for our application, the CTO was approved, but until one day before the day, I still doesn't have an access to our new VDC. In the next day, the CTO arrange a meeting with ZettaGrid team and we order the services that fit with our infrastructure. After that I must running quickly to setup all of everything that match with the infrastructure, installing Centos, configuring development package, configuring database, also file management and firewall, and many more.
Within 3 hours the entire infrastructure has been ready, alhamdulillah. But when the application deployed, we still have a problem in our website application (before we're migrating to the new provider, we are using httpd later we are moving to use nginx). Caused by file permission of the frameworks language, also problem with SELinux. Make me frustrated for a while, but soon could be fixed by quick and dirty solution. Later we still have many issue, like the connection problem into our database from office, SSL redirection, PHP-FPM with nginx and so on. In a week I learn a lot a linux command line, actually this is very interesting for me. One by one issue has been resolved and then I move to increase the performance of our system by adding Load-Balancer scheme, creating bash script for easy deployment (we still doesn't use CI/CD btw).
Now, our system better than two months ago, we have made it automatic in archiving, updating package, also updating container (we're using docker). We also separate request from public, office and integration use with 3rd party for better performance. Moving fast, we are on the stage that we need to integrate our system with the other 3rd party beside add more features to support business needs. And I got assigned to maintain all of the entire environment (development, staging, production) also git with git-workflow for better release to follow semantic versioning, also integrating with 3rd party (mostly with Bank).
At the end, I take a risk when we're on going to integrate (Host to Host) with bank by separate the module into a small services with Vert.x. Because learn from the past when we're integrating with Payment Gateway, we have some limitation when communicate with RestApi, so I decided to build the integration services with Vert.x. There're another story in this part which I would publish on another post.

12 November 2018

[FIXED] Java MacOS Hostname

During the development of Caltic, there is a strange from internal logger of Vert.x (our toolkit to build Caltic Systems). There're some warning log indicate something happen abnormally. See below:
[INFO] Nov 08, 2018 5:24:50 PM io.vertx.core.impl.launcher.commands.Watcher
[INFO] INFO: Watched paths: [/Users/myrepublic/devel/caltic/caltic-train/target/classes]
[INFO] Nov 08, 2018 5:24:50 PM io.vertx.core.impl.launcher.commands.Watcher
[INFO] INFO: Starting the vert.x application in redeploy mode
[INFO] Starting vert.x application...
[INFO] 18b3e3d3-043d-4f63-b8aa-52d018554b6a-redeploy
[INFO] Nov 08, 2018 5:24:53 PM io.vertx.core.impl.BlockedThreadChecker
[INFO] WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2858 ms, time limit is 2000
[INFO] Nov 08, 2018 5:24:54 PM io.vertx.core.impl.BlockedThreadChecker
[INFO] WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 3859 ms, time limit is 2000
[INFO] Nov 08, 2018 5:24:55 PM io.vertx.core.impl.BlockedThreadChecker
[INFO] WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 4863 ms, time limit is 2000
[INFO] Nov 08, 2018 5:24:58 PM io.vertx.core.impl.BlockedThreadChecker
[INFO] WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2520 ms, time limit is 2000
[INFO] Nov 08, 2018 5:24:59 PM io.vertx.core.impl.BlockedThreadChecker
[INFO] WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 3524 ms, time limit is 2000
[INFO] Nov 08, 2018 5:25:00 PM io.vertx.core.impl.BlockedThreadChecker
[INFO] WARNING: Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 4524 ms, time limit is 2000
It was happen when I upgraded MacOS Sierra to MacOS Mojave and along with it also upgraded JDK 1.8_171 to 1.8_181. For several weeks, I think it's not cause serious problem since the logs doesn't appear when the apps deployed to Heroku.
Later when I read Vert.x Core Manual, I carried to actor model and one of them that using actor model is Akka. Yeah, one year ago when I did research on Play Framework, the web framework for Java and Scala, I also found Akka, but I didn't found Akka deeper, when read one by one topic in Play. But now I understand that Play is only one implementation module (Akka Http) from many modules that Akka have, like Akka Actors, Akka Streams, etc. Back to the topic, when I tried the Lagom Framework, the new brand from Lightbend for microservice purpose, I also notified that something happen when starting Http Server. In the warning logs from Lagom, there is an url recommendation to solve this issue, here. Then the investigation is started.
After cloned git repository from the url above and run it on my machine, I got:
~$ java -jar bin/inetTester.jar
Calling the hostname resolution method...
Method called, hostname Heru-MacBook-Pro.local, elapsed time: 5011 (ms)
~$
Wow it's take a time 5 sec to getting the hostname, too long. Based on the url from Lagom logs, the proposed solution is fixed file /etc/hosts. The file should be consists of:
127.0.0.1   localhost Heru-MacBook-Pro.local
::1         localhost Heru-MacBook-Pro.local
The format above is <ip><localhost><hostname>. The hostname is grabbed from inetTester logs above, in my case is Heru-MacBook-Pro.local. Actually I'm using VIM to edit this file, but if you don't familiar with them, you could using echo.
~$ sudo echo "127.0.0.1 localhost Heru-MacBook-Pro.local" > /etc/hosts && sudo echo "::1 localhost Heru-MacBook-Pro.local" >> /etc/hosts
When password is asked, type on the terminal then enter. After you running these command, you could verify with cat command.
~$ cat /etc/host
127.0.0.1   localhost Heru-MacBook-Pro.local
::1         localhost Heru-MacBook-Pro.local
~$
Ok the proposed solution has been run, now time to verify. We can move to inetTester app again, run it:
~$ java -jar bin/inetTester.jar 
Calling the hostname resolution method...
Method called, hostname Heru-MacBook-Pro.local, elapsed time: 8 (ms)
~$
Amazing, you see it? It's only take time 8 mili-sec, so faster than before (5 sec). Now let's try the caltic app.
Vert.x Application Log
No blocked thread anymore. Since the threshold of each event loop from vertx is 2000 ms, so if your code is taken more time to be processed than the threshold, the warning log should be appeared, otherwise. Case Closed!