Blog

Journal

My CloudNativeCon + KubeCon ’17 Experience

I was awarded a diversity scholarship to go to CloudNativeCon + Kubecon in Austin. I was selected in the second round, only 30 scholarships were offered at first. However, thanks to the efforts of the diversity committee they were able to offer 73 more scholarships. I’m very grateful for the diligence of these ladies to extend this amazing opportunity to more people.

Community

I’ve always found the Kubernetes community to be open. Good people like the Service Catalog SIG who had some much patience guiding me towards my first contribution. Or the kubeadm folks that helped me troubleshoot when I got stuck building my first bare-metal cluster(setenforce 0 on reboot, never forget). Kubecon was no exception. Since Day 0 I found myself talking to experts as well as beginners. Everybody I approached was willing to talk about their ideas and work, as well as listen to mine and offer advice. I was able to meet people that have helped me in my journey-shout out to Nikhita and Aaron-as well as meeting some new friends, like Jackie, Jonathan, and many others.

Service Mesh: Where the winds are flowing?

I was able to attend an Istio Workshop on Tuesday. I had heard about Istio and service mesh before but here was when I finally was able to grasp the concept and understand the problems it looks to solve. Overall, I perceived a lot of enthusiasm around Istio, it looks like is going to be a great year for Istio as more applications migrate to k8s and use it.

Lightning talks

I attended the lightning talks to see Nikhita’s talk about contributing to k8s(great talk by the way). I stayed for the rest and got many ideas from them. As Sarah, the MC, puts it “lighting talks are meant to spark ideas”. Special mention to Jonathan Rippy and his “Watch This!” talk. Of all the whole conference this was my favorite talk because I felt connected to due to my own experience putting to together a cluster with kubeadm.

Austin, TX, Winter Wonderland


Sooooo, it snowed in Austin. Not my first time seeing snow, but I’ve been living in the Dominican Republic for the past 4 years, so my body was definitely not ready for it!

This time around I got me some Texas BBQ at Cooper’s.
Also had the chance to visit Graffiti Park.

Conclusion

This conference has been a great step in my career. I was able to learn a lot about cloud and k8s, grow my network, and have a better idea of what I need to learn next to keep growing. I hope I can repeat it sometime in the future, maybe even put together a talk to share my own knowledge and experience.

Journal

Journal: Latter Half of October ’17

So here is an update on what I’ve been doing in the latter half of October.

Prometheus

I’m finally playing around with Prometheus and monitoring which is something I wanted to do for a long time. I’ve started by creating a 5 node cluster on GKE and then following a simple guide, Kubernetes monitoring with Prometheus in 15 minutes by Giancarlo Rubio. Afterwards, I watched two talks about the subject Monitoring, the Prometheus Way by Julius Volz, one of the founders of Prometheus. Also, End to end monitoring with the Prometheus Operator by Frederic Branczyk, he works for CoreOS and is one of the maintainers of the Prometheus Operator. I’ve learned a lot from both talks, now I plan to dive deeper into monitoring. As a side note, I’ve also watched Taking the Helm: Delivering Kubernetes-Native Applications by Michelle Noorali, Matt Butcher, and Adnan Abdulhussein.  I watched it because Helm is a very important project, in my opinion, and I used it to install the Prometheus operator.

Service-Catalog

Last time, I mentioned I was reading up on the k8s API since I was working on a ticket. Here is the Pull Request. This one was basically Bash scripting and Makefiles, however, I did learn a lot!

Neovim

So I’ve been busy learning VIM, however, I decided to start using Neovim instead. Is a drop-in replacement for VIM that tries to keep what’s good and add new features. The same configuration works. I’ve also created a dotfiles repo to keep track of my configuration. Have a look!

Conclusion

That’s about it for these last two weeks. I’m planning to stick with the same subjects going into November. Catch you later!

Journal

September & October ’17: kubeadm, K8s API, VIM, and a little Housekeeping

I’m trying a new approach for my documenting process. Namely:

  • Shorter posts on the blog.
  • More tweets to share more of the day-to-day.
  • Walkthrough guides will now be versioned(as they should) on GitHub.

Without further ado here’s what I’ve been up to:

Bare Metal & kubeadm

I picked up kubeadm and build a bare metal Kubernetes cluster. I wrote a guide about it, you can find it on GitHub.

I learned so much on this project:

  • I realized that building a cluster is not trivial(yet).
  • There are many tradeoffs to make.
  • Improved my Kubernetes debugging skills since I had to look at the kubelet, docker, pods, nodes, etc.

Kubernetes API

I’m working on a ticket for service catalog that requires that I have a deeper understanding of k8s architecture and APIs. So, I took some time to reread and comprehend the concepts and API conventions. Similarly, I’ve been looking at the build process for both k8s and service catalog.

Housekeeping

Perhaps you’ve noticed some changes in the blog. There is a new theme and the coffee beans header is gone. My domain now points to the (shorter)bio and I set up a simple email forwarding following this guide. This is part of my commitment to keep documenting. I plan to keep adding features and styling as time goes on.

VIM

I’m also learning VIM. Two months ago, I taught myself how to touch type with gtypist as part of the process to learn VIM properly. Now I’m picking up VIM using vimtutor and following advice from this guide.

Conclusion

Busy month. For the near future, I want to explore Helm and Prometheus on my new cluster. Also, I plan to keep contributing to the Kubernetes project and learning VIM. Catch you on the next one!

Journal

My Hashiconf ’17 Experience

Last week I had the opportunity to attend Hashiconf ‘17 in Austin, Texas. I loved the atmosphere at the event and the city. I met some interesting people and learned how they use Hashicorp products to solve their problems. Here is a brief summary of my favorite talks and experiences.

How to Build Reusable, Composable, Battle-tested Terraform Modules by Yevgeniy Brikman

The Terraform series of post I’ve talked about in my last post were written by Yevgeniy. So, of course, I attended his talk to learn more about Terraform. Thankfully for me, the talk centered around modules which I’ve not had the chance to study in depth.

Your Secret’s Safe with Me – Securing Container Secrets with Vault by Liz Rice

I watched one of Liz’s previous talks on containers and learned a lot. This talk was also very instructive and served as an introduction to secrets for me. Most important lesson I learned was not to put secrets in environment variables and why. Also learned how cool Hashicorp’s Vault is and the status of secrets managements in K8s, Nomad, and Docker Compose.

Health-Checking as a Service with Serf by Lorenzo Saino

Lorenzo had a tall order, close day 1;  he delivered big time. His talk was about how fastly designed their health-checking system under constraints using Hashicorp’s Serf. My favorite part was when he explained how they use digital signal processing and control theory in the system. Turns out each node health check data is, in fact, a discrete signal! This talk really hit home and brought me back to my college days as a computer engineer.

#Hashinetes by Kelsey Hightower

Kelsey delivered a great keynote on Day 2. He talked about Kubernetes + Hashicorp(Nomad, Consul, Vault) = Hashinetes. At some point, he even used voice commands to control the cluster.  Two schedulers! Extra dope.

Practical Distributed Consensus using HashiCorp/raft by James Nugent

I’ve been interested in studying Raft for a while. This talk was a great intro and gave me a new idea on how to approach the learning experience. Rather than implementing the algorithm myself, I could use the already existing Hashicorp library to learn in a practical approach.

Digital Ocean Brunch

On Thursday morning I attended a brunch sponsored by DigitalOcean to conference attendees and local users of their product. I enjoyed this event as it allowed me to unwind and connect with the local developer scene and people from the company.

Austin, Texas

It was my first time in Austin so after the conference, I stayed two more nights to explore. I visited the Blanton Museum of Art and was impressed by the variety of the permanent collection. They have art from North America, Latin America, and Europe, from various periods. I learned some Texas history on a tour of the capitol. At sunset, I was able to see the see the bats under the Ann W. Richards Congress Avenue Bridge, over a million bats!

Conclusion

I had a great conference experience and a great mini-vacation in Austin. I was able to learn new things and meet interesting people. I feel refreshed and inspired to keep diving deeper and learning new things. So stay put and expect more posts!

DevOps

Creating a Simple Cluster with Terraform

Since my last post I’ve been reading about Consul. Also, I found an interesting book: Site Reliability Engineering by Google. So far I’ve read the intro and first chapter and I’m captivated. I want to take my career into SRE/DevOps and this book provides good insight. However, this post is about Terraform, I found a great blog and learned more about it.

The Case for Terraform

I found a great series of posts by the engineers at Gruntwork called A Comprehensive Guide to Terraform. In the first post, the author talks about their choice to use Terraform. Interestingly, the article describes trade-offs between Terraform and other choices as:

  • Configuration Management vs Orchestration
  • Mutable Infrastructure vs Immutable Infrastructure
  • Procedural vs Declarative
  • Client/Server Architecture vs Client-Only Architecture

I really liked this approach because I learned about how other technologies approach the IAC(Infrastructure As Code) problem in comparison with Terraform. Technologies such as Chef, Puppet, Ansible,etc., are now a clearer in how they work and where they might be the right solution.

The Guide That I found After My Initial Post

The second post of the series is an i ntro guide to Terraform just like my previous post. Yet I learned new things while going through it. First new thing I learned, is to set up a new AWS user with limited power instead of using the root account that is created by default. This is described further in the AWS IAM(Identity and Access Management) Best Practices guide. I had to delete my root key as a security measure and then create a new user. See Creating Your First IAM Admin User and Group. Also, I learned how to name an AWS instance with a ‘tag’.

After creating an initial instance, the guide goes further by deploying a simple web server. First, we make use of an EC2’s User Data to run a simple script at launch that starts our server. Then we create a new resource, a security group, to allow our server to be reached from any IP through port 8080. After$terraform apply we can hit our web server with $curl http://<EC2_INSTANCE_PUBLIC_IP>:8080. The public IP address is set as an output in the tf file.

The guide goes on to create a cluster of web servers using an AWS Auto Scaling Group(ASG), which takes care of starting/restarting instances, monitoring health, and adjusting scale based on demand. We start by replacing the “aws_instance” with an “aws_launch_configuration”. This new resource configuration is similar to the aws_instance with the addition of the “lifecycle” block. Alas, we must add this new block to all resources on which the new resource depends on. In this case, we must add it to the aws_security_group. Then we add the ASG resource as “aws_autoscaling_group”. Finally, we use a data_source to fetch all the availability zones from AWS and use them within our ASG as availability_zones = [“${data.aws_availability_zones.all.names}”]. This new variable allows our EC2 instances to be deployed in different(isolated) data centers.

We need two more resources. First, since we have multiple instances running we need a load balancer to distribute the traffic to our instances. For this example, we use AWS very own load balancer, Elastic Load Balancer(ELB). As part of the configuration, we add a “listener” to tell the load balancer how to route incoming requests. In this case, we simply route port 80(HTTP) on the ELB to our instances’ port 8080(set in the variable server_port). The second resource is a new security group to enable traffic to the load balancer. The guide goes on to add “health_check” on the load balancer. This requires outbound traffic to be enabled as an ‘egress’ on the security group. Finally, we must register the instances with the ELB everytime a new instance is created. We do this by adding health_check_type = “ELB” to the ASG configuration. Finally, we add the ELB DNS as output to be able to test our deployment. Hit $terraform plan and if everything looks good $terraform apply. As the resources are being created can check the console for the resources. Once the instances are running we can $curl http://<elb_dns_name> and expect “Hello,  Word” back.

Conclusion

I learned  more about Terraform and AWS with these two posts. I’m impressed by the way we could deploy a simple cluster with a few lines of code thanks to Terraform and AWS.  We even had health checks! Well, I’m wrapping up this post at 30,000(28,014)ft feet in the air on my way to Austin for Hashiconf. Hopefully, I’ll learn more about Terraform and the other solutions from Hashicorp as well as meeting some smart people. Catch you on the next one!

DevOps

Creating Infrastructure with Terraform

I’m attending HashiConf ‘17, an annual tech conference from HashiCorp. I came across HashiCorp sometime ago when I was playing around with Vagrant, a neat tool for creating and managing dev environments using virtual machines. I became aware of the conference thanks to Twitter and reached out to the organizers asking if there was any financial aid or scholarships. Lucky for me, I was offered a complimentary ticket to attend. So here I come HashiConf ‘17!

Terraform

Since I’m attending the conference I want to become familiar with other HashiCorp products. I’m going to start with Terraform following the intro guide on Terraform.io. According to the guide, Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. It can manage popular services providers as well as custom solutions.

Install & Build

First, we need to install terraform. The guide provides a link to the binaries but I decided to install it using Homebrew.

$brew install terraform
$terraform version #verify it was installed properly

After the installation, we start to build basic infrastructure on AWS. First, we create a Terraform configuration file(.tf) and copy and paste our access keys for now. We can figure out the keys on AWS’s IAM console. I created a pair easily by clicking on “Create new access key”. Then, we do $terraform init which basically is the “mise en place” or the initial configuration to start building the infrastructure. We can see how Terraform plans to create(or modify) the infrastructure with: $terraform plan. If we get no errors back we can go ahead and proceed with apply: $terraform apply. Which creates the instance. We can then do $terraform show to check the state. The guide asks that we check the console to confirm the instance was created. However, this was not the case for me! I tried looking for any report of this error online but found nothing. Similarly, I couldn’t find the ami “ami-2757f631”, my guess is that the instance is not provided by Amazon anymore. The fine print in the guide states that:

Note: The above configuration is designed to work on
most EC2 accounts, with access to a default VPC. 
For EC2 Classic users, please use t1.micro 
for instance_type, and ami-408c7f28for the ami. 
If you use a region other than us-east-1 then you will 
need to choose an AMI in that region as AMI IDs are 
region specific.

Some new terms for me. Amazon defines VPC(Virtual Private Cloud) as a virtual network that resembles the network in a data center. Also, EC3 Classic only applies to accounts created before 2013-12-04. Since the next chapter in the guide is about deleting and modifying infrastructure, I’m going to keep going to see if changing the ami solves the issue.

Change the Infrastructure

The guide wants us to modify the ami and see how Terraform handles the change. It asks to change to ami-b374d5a5. However, after looking for this ami I cannot find any reference on the ami section of the EC2 dashboard. I’m going to change it to ami-da786da3 which is Suse Enterprise 12. I chose this image from the ‘launch instance’ menu. I did $terraform apply and I got the following error:

Error applying plan:
1 error(s) occurred:
* aws_instance.example: 1 error(s) occurred:
* aws_instance.example: Error launching source instance: InvalidAMIID.NotFound: The image id '[ami-da786da3]' does not exist
        status code: 400, request id: ccf07eee-c5c9-4204-b308-0614e2915af8

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Which points to the error being my fault. Terraform does correctly give an error message when a wrong image is used. Though in hindsight, I think it would be better to know in the plan stage not after the previous instance was destroyed. Now, I have two doubts: (1) What console am I supposed to find the instance? (2) Where are the appropriate images?

I’m going to change the ami to what the guide suggest: ami-b374d5a5.  Then $terraform plan & $terraform apply. No errors.

I found the issue! Turns out, I was in the wrong region on the console! I was looking at the west region, where the blog resides. The guide, however, is working in the East region. I’m going to try to modify the configuration with a Suse image from this region. AMI being ami-8fac8399.

aws_instance.example: Creation complete after 51s (ID: i-0a3da5f272e78ec90)

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

Success!

Destroy

Next step on the guide is to destroy. We can see what’s going to be destroyed with $terraform plan -destroy. Then we can destroy with: $terraform destroy after which we get prompted to confirm we want to delete the resources.

Resource Dependencies & Provision

In this section, we are introduced to configurations with multiple resources and dependencies between resources. We start by assigning an elastic IP to the EC2 instance by adding an “aws_ip” resource to the configuration. In this new resource we interpolate the instance ID using a reference:instance = “${aws_instance.example.id}” . When we do $terraform plan we can still see the value since its interpolated once the resource is created. After $terraform apply we notice an order in the creation of the resources, first the instance then the elastic IP. This is an implicit dependency, terraform takes care of this automatically. We can also create explicit dependencies with the “depends_on” parameter.

We run $terraform destroy and move onto the next part of the guide, Provision. Provisioners are introduced as a way to do some initial setup on instances, be it running scripts or a configuration management tool. As an example, we add a provisioner to the example instance which runs the command “echo ${aws_instance.example.public_ip} > ip_address.txt” when the instance is created. If the provisioning fails, the resource is marked as tainted and destroyed and recreated on the next execution plan. Finally, there are also destroy provisioners that are executed during the destroy phase.

Input & Output Variables

In this section we’re introduced to variables as a way to make configurations shareable & version controlled. As an example, we create variables for the access keys as they are hardcoded at the moment. First we create a file(variables.tf) to declare our variables and replace these names in the configuration. Now when we $terraform plan we are asked for the values of these variables. We can assign  the values as command-line flags, set them in a separate file or set them in environment variables.  We can also create lists and maps. For the example we create a map with amis for each region. Finally, we can also create output variables which serve as a way to organize output data that can be queried by the user.

Modules & Remote Backends

Since modules require non-free tier usage of AWS I’m going to skip the hands-on part of the tutorial. Modules are a way to make components reusable and help organize the workflow. To me, the most interesting part of the modules is the source field in the module declaration. In this example, we create a Consul cluster from a configuration that is saved as part of the GitHub project: source = "github.com/hashicorp/consul/terraform/aws". Finally, remote backends are a way to handle terraform in production. As the guide says, remote backends are used to run terraform remotely and store a master Terraform state remotely.

Conclusion

I learn a few important lessons from working through this tutorial:

  • I got an overview of a cool way to manage all this infrastructure in a clever and organized way.
  • Read and Understand the fine print in guides! Or anywhere for that matter
  • I learned more about the features of AWS and to be aware of the regions.

Looking forward to the conference and learning about Terraform. Catch you on the next one!

development, Journal

July & August 2017: My first PRs

So last time I mentioned I was looking to contribute to the Kubernetes project. After an initially steep learning curve, I’ve managed to get two PRs approved on Service Catalog, a Kubernetes incubator project. Here’s how that happened.

How I got started

I was looking for a good place to start on the k8s project. Coincidentally, I came across a tweet by @TheNikhita and decided to reach out and ask her about her experience and tips contributing to k8s. She suggested that I start by setting up the dev environment(see running locally and development ) and reading the API convention. Both of which are great resources.

I also read the k8s community README. Here they suggested that the first step to contribute is to join a SIG(Special Interest Group) to figure out what’s going on and ask for a good place to start.

Service Catalog

Service Catalog is a Special Interest Group(SIG) that is looking to implement the Open Broker API. I got interested because is a new project, alpha about to go beta, and is its own subsystem, with controllers, API server, etc. I believe is a great opportunity to get experience on an early stage project and help it grow. So I joined the meeting (Mondays 20:00 UTC) and after I reached out to one of the leads @cheddarmint for tips on how to get started. He suggested following the walkthrough to get the catalog deployed on a k8s cluster. I was able to deploy the catalog without major drawbacks.

My first approved PR

On the meeting on June 26, @arschles mentioned he was going to add some new test and refactor some old tests, that it would be a great place to start. So I reached out to him and he suggested I worked on issue 860. So I called #dibs on the ticket.

I started by following the workflow from the k8s development guide. Basically,

  1. Fork in the cloud
  2. Clone locally the fork
  3. Make your changes on a branch
  4. Keep in sync with upstream
  5. Push to your fork
  6. PR to upstream

However, I made a critical mistake in the process. I cloned to $GOPATH/src/github.com/crmejia/ instead of $GOPATH/src/github.com/Kubernetes-incubator. This simple mistake made it impossible for me to compile locally and run the test faster. For a while, I was running the test on docker until I realized my mistake and correctly set up my clone. I added the process to the svc-ca guide, my first PR.

Unit Testing & Go

Now for issue 860, I was tasked with testing an important testing component a fake rest client($GOPATH/src/github.com/Kubernetes-incubator/service-catalog/pkg/rest/core/fake/rest_client.go). Which encapsulates TPR based storage functionality within a fake client.

But how to go about testing in Go?I found a good guide covering basic testing. So I started by testing the main data structure of the client, NamespacedStorage, which is a map of namespaces to types. These types are themselves a custom TypedStorage mapping types to ObjectStorage, which is mapping object names to k8s runtime.Objects. So basically a map within a map within a map. I tested the methods defined on NamedspacedStorage: Get, Getlist, Set, and Delete. I did not find this test very complicated since I have experience with data-structures.

It wouldn’t be a rest client if it couldn’t REST. So there is a RESTClient struct defined in rest_client.go that extends the fake client defined in the package k8s.io/client-go/rest/fake and adds a NamedspacedStorage, a Watcher, and a MetadataAccessor. Also. there is a private responseWriter struct defined with its own methods, testing these methods wasn’t complicated either.

Now the part I got stuck was when testing the RESTClient itself. At first, I was trying to test the handlers directly but it didn’t work since the router was not populated. After some time wondering why I got no objects out of populated storage, I figured that I needed a working router and that populating manually would be naive as it would be duplicating the rest client’s code. So I read up on gorilla/mux, the router used by the client, and started testing the handlers using the router. I had the good fortune to come across a video justforfunc #16: unit testing HTTP servers by @francesc that gives a good insight on how the test an HTTP server and how to use test tables and subtests. Sometimes I would get stuck on some service catalog details, thankfully the people over at the #sig-service-catalog in the k8s slack would clear up the confusion every time.

Finally, after much trial and much learning, I completed the PR. I had to close the old one as I had signed the CLA(contributor license agreement) with a different email than the one I had for Github. So I had to make a new PR. However, this was also part of the reason I understood how to setup the clone properly.

Conclusion

I was able to complete one of my short-term career goals, contribute to an OSS project. Is not a breakthrough contribution but it proves to me that I can do it. I’m nowhere near an expert in k8s but I took another step in the right direction. Going forward I want to keep contributing and learning while documenting. Catch you on the next post!

Journal

June 2017 Journal: What I’ve Been up to

Howdy folks! It’s been a while. I’ve kept myself busy this past month learning about different things in kubernetes and distributed systems. I’ve document most of it but I felt that nothing was worth a post that is until I found this article by Sam Jarman titled “Online Presence”. The article talks about, you guessed it, online presence in its various forms(blogs, vlogs, github, etc) but what really caught my attention was a linked video: Gary Vaynerchuk’s Document, Don’t Create. After the video Sam gives some tips on what to document and how to document as a developer. That’s what I’m going to do going forward: document my learning process, log my journey, tell my story.

Kubernetes the Hard Way

Right after setting up the blog I was asking the recurring “Where to go next”. I decided to complete Kelsey Hightower’s Kubernetes The Hard Way guide. In this guide we use GCP and since I spend all the credit in my main account I had to set up another trial in my old account. This time I’ll be more careful with clusters. Overall, the guide was straight-forward. I learned more about components and how many of them you need to bootstrap a cluster.

Running Workloads in Kubernetes

I learned about application patterns and how they’re addressed in k8s from Janet Kuo’s Running Workloads in Kubernetes. Turns out my blog project follows a “stateless pattern”. But there are also other patterns such as stateful, daemons, and batch. We do not set up datastore cluster the same way we schedule tasks. I will set up some quick projects with these patterns later on.

The Log: What every software engineer should know about real-time data’s unifying abstraction

I cannot for the life of me remember where I found it, I guess that’s another good reason for  documenting often. I know it was in another article that referenced it as if you take away anything from here let it be reading this excellent article. So I did. Before reading, my concept of logging was more of an application log for humans to read. The author puts it best:

But before we get too far let me clarify something that is a bit confusing. Every programmer is familiar with another definition of logging—the unstructured error messages or trace info an application might write out to a local file using syslog or log4j. For clarity I will call this “application logging”. The application log is a degenerative form of the log concept I am describing. The biggest difference is that text logs are meant to be primarily for humans to read and the “journal” or “data logs” I’m describing are built for programmatic access.

So the article did teach me a better important lesson. Now when I have to design a system or a service I will consider a log centric approach. As a plus, I understood some of the ideas behind Apache kafka. So what are you waiting for go read The Log: What every software engineer should know about real-time data’s unifying abstraction.

Consensus -> Orchestration -> Raft

So I learned about consensus in Coursera’s Cloud Computing Concepts, Part 1 and then I  came across this talk about orchestration by Laura Frank. I really like it since it painted a picture of where consensus comes in for cluster orchestration. In k8s’ case in comes in etcd’s choice of the raft algorithm. My favorite part of the algorithm though is the website because it gathers all the key information you need to understand it: papers, talks, courses, where to ask questions, and real-world examples. I found some interesting assignments in the courses listed to implement raft. I hope to give it a go further down the line but for now let me stay on k8s.

Where to Next?

One of my short-term career goals is to get involved in a OSS project and contribute. So after completing the guide and reading the articles I decided I want to contribute to k8s. I’ve already started working towards that goal. I found some good people willing to help out and a very interesting kubernetes SIG(Special Interest Group) but that’s story for another post. I want to advance further before sharing my experience. Sort of not to be jinxed. I’ll be documenting however. See you then.

Blog Meta

AWS EC2 + Bitnami and HTTPS w/Let’s Encrypt

I ran out of credit for Google Cloud. Turns out running a cluster is quite expensive. For the sake of having a blog while I keep learning about kubernetes and other technologies I decided to go with a simpler(and cheaper) approach to blogging.

Enter AWS + Bitnami

I chose to start AWS free trial and create a blog using the prepacked Bitnami solution. This deployment runs on a good ol’ VM and is cheap. Also, I get the chance to familiarize myself with the AWS stack which is useful since I could create a k8s cluster down the line and play with it.

Bitnami pre-packs WordPress, MariaDB, Apache HTTP Server, and some neat tools to facilitate a wordpress installation and setup. It’s a straightforward stack to set up and there is some fantastic documentation available.

I reached out to a colleague regarding my domain issue. Turns out I was using a forward on my A record hence the blog’s IP showed instead of the domain. After modifying the record I was able to have my domain show up.

HTTPS with Let’s Encrypt

Now that the blog is up I can work to enable HTTPS which is something I’ve been looking to do for a while. First, we need to get a certificate from a Certificate Authority(CA) which can be expensive see here and here. Fortunately, we can get a free certificate from Let’s Encrypt that provides just what we need to enable HTTPS.

There are straightforward instructions on how to get a certificate using EFF’s Certbot if you have shell access. For Bitnami the process is a bit different since there are tools to manage certificates. We still need to install the Certbot but the certificates need to be put in the corresponding directories within the bitnami app. Following the guide from Bitnami docs here are my steps:

First we need to install git and certbot. In the AWS instance:

$sudo sudo apt-get install git
$cd /tmp
$git clone https://github.com/certbot/certbot
$cd cerbot
$./certbot-auto

At the end of the installation we get the message

Failed to find executable apache2ctl in PATH: /opt/bitnami/varnish/bin:/opt/bitnami/sqlite/bin:/opt/bitnami/php/bin:/opt/bitnami/mysql/bin:/opt/bitnami/apache2/bin:/opt/bitnami/common/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin

Certbot doesn’t know how to automatically configure the web server on this system. However, it can still get a certificate for you. Please run “certbot-auto certonly” to do so. You’ll need to manually configure your web server to use the resulting certificate.

Run again, this time:

$./certbot-auto certonly --webroot -w /opt/bitnami/apps/wordpress/htdocs/ -d crismar.me

After accepting the terms and providing our email we get a certificate! Now we must install it on the server. The guide we are following doesn’t state this but the apache server on this wordpress pack has a dummy cert that we need to replace with ours. The guide creates symbolic links to the certs instead of copying them, my guess is that this is to automate the cert renew process since the certs expire every 3 months. 

$cd /opt/bitnami/apache2/conf/
$rm server.crt
$rm server.key
$sudo ln -s /etc/letsencrypt/live/crismar.me/fullchain.pem server.crt
$sudo ln -s /etc/letsencrypt/live/crismar.me/privkey.pem server.key

Then add the following lines to the  /opt/bitnami/apps/wordpress/conf/httpd-prefix.conf to force https:

RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [R,L]

And once again modify the wp config file to point to https:

$vim /opt/bitnami/apps/wordpress/htdocs/wp-config.php define('WP_SITEURL', 'https://crismar.me/');
define('WP_HOME', 'https://crismar.me/');

A small digression. In the previous post Let There Be a Domain, I explained how I broke the blog by changing these variables. At the time I didn’t quite understand what went wrong but now I finally get it. Basically, the goal of these variables is to generate the URLs of the blog. That is all the links to all the pages and posts. In our case we want these links to be a domain rather than an IP. Turns out since my DNS was (mis)configured to forward I had a loop between my server and the DNS server that never resolved. My domain pointed to the IP but WordPress pointed to my domain. All in all, now I have a better understanding of DNS and servers and how to connect to pods in a cluster.

Finally, lets restart Apache httpd:

$sudo /opt/bitnami/ctlscript.sh restart apache

And the blog should be all green locked! Almost. My login page and admin console are but some pages are not.

Turns out, I have some mixed content

Mixed Content: The page at 'https://crismar.me/2017/04/16/blog-day-1-site-unavailable/' 
was loaded over HTTPS, but requested an insecure image 
'http://i.imgur.com/oXki5DH.png'. This content should 
also be served over HTTPS.

In other words, my resources have to be HTTPS too! Let’s edit the post(s) and change that. That did it. Green!

Conclusion

This time I had the chance to familiarize myself with AWS and to explore how some companies such as Bitnami work to make setup and installation simple. Also, I got around to setting up HTTPS and Certificates and learned what it takes to make a server secure. Now that the blog is rolling(in a secure HTTPS way I must add) and stable. I plan to go back to kubernetes and create a cluster from scratch. Until then!

Blog Meta

Let There be a Domain

Now that the blog is back up finally we can finally get around to setting up a domain. First thing we would need is a static IP to point the domain to. So let’s do.

Static IP on GKE for a Kubernetes Service

I did not find an official guide on how to do this. Fortunately, I found a blog post and a Stack Overflow answer that explains how to accomplish this. In summary, there are two ways (1) reserve a static IP address and then assign it to the load balancer or (2) change an already existing load balancer IP from ephemeral to static. Both are done from the GCP Networking > External IP addresses dashboard. Since the blog is already UP, I’m going to switch the service IP to static. Now we must add a loadBalancerIP: 10.10.10.10 field to our service deployment. That way if we need to redeploy k8s knows where it needs to point.

Let There be a Domain

I thought about it for a while and looked around on what would work as the blog name. Since is a software/technology blog I notice some of the most popular developer’s blog use name and/or last name. So it made sense to use my name or something rather simple as the blog name. I found a short blog post going over some tips on what makes a good name. The tip: a blog name should be spellable caught my attention because my last name is not spelled(in english) like it sounds. So I sticked to my first name. I tried to get ramsir.com, my name spelled backwards, since the .com would add the C and my other initials(Mejia, Ozuna). Alas, it was taken as well as ramsirc. I  settled on crismar.me and bought the domain on godaddy with default setup: forward with masking to the server IP. Oh and I had to spend some time reading about DNS to freshen my memory. Julia Evans has some cool zines on networking and talks briefly about DNS.

Unresponsive blog

The blog is down again. I changed two settings: WordPress Address URL and Site Address URL and broke the blog. I figured I should use my domain here instead of the IP but the changes broke the site.

Hardcoding URLs on the config file

I found a guide that explains how to manually hard code these settings in the main config file(wp_config.php). The challenge is how to set these values in a cluster setup, persistent volumes attached to a pod. I found another guide that explains the process. Here’s the process step by step:

    1. From GCP, ssh into the VM of the node using the disk. We can see the disks from https://console.cloud.google.com/compute/disks We click on the VM of the node we are redirected to the VM compute instance.
    2. Here we can click on the SSH button in the top left corner and connect to a fully function bash terminal hassle-free:
    3. Once we get access we proceed to mount the persistent disk.

$lsblk #list the disks
$sudo mkdir -p /mnt/disks/disk1
$sudo mount -o discard,defaults /dev/sdc /mnt/disks/disk1/
$cd /mnt/disks/disk1
$ls

    1. Make a backup of the the wp_config.php file,then open it in your favorite editor and add the lines below to set the url to be IP.

$cp wp-config.php ~
$sudo vim wp-config.php

define('WP_HOME','http://10.0.0.1');
define('WP_SITEURL','http://10.0.0.1');

Back online! However, the settings are hardcoded and we are not able to change them in the admin dashboard. It would be better to connect to the DB and set the values back. So let’s do it!

Setting the URL through the DB

Connecting to the DB means connecting to the mysql container running inside a pod. Just like docker allows us to run commands on a container using docker run, in Kubernetes we can use kubectl exec to run commands on a pod’s container(s). In our case the MySQL pod has only one container but if we had more we could specify the container by using the -c option. Here’s are the steps:

    1. connect to the DB

$kubectl get pods #basically get the pod name
$kubectl exec wordpress-mysql-2569670970-f1grp -- env #copy the mysql password
$kubectl exec wordpress-mysql-2569670970-f1grp -- mysql -p

By this point we are inside the mysql cli. Notice that I did not set the password as I was not able to use the environment variable MYSQL_ROOT_PASSWORD as an argument just like docker run command does by using quotes. I did a little search and found this issue which describes problems with quotes. There is a PR on github trying to address the issue but is on hold as it broke some other functionality. So for the sake of brevity I decided to copy the env and paste it into mysql.

    1. Update the wp_options table

In section 1.4 of the wp guide there is a brief explanation on how to change the URLs in the DB. It does so by accessing phpMyAdmin so the guide is only useful for us to get the table and fields that we need to change. Those are table wp_options and fields siteurl and home. So back in the mysql cli we run:

mysql> use wordpress;  #set the db. Notice the ;
mysql> select * from wp_options where option_name = 'siteurl' or option_name = 'home';
mysql> update wp_options set option_value = 'http://104.196.161.37' where option_name = 'siteurl' or option_name = 'home';
mysql> \exit

Done. Now we can go back to the node VM, mount the wordpress persistent disk and remove the hardcoded values.

Going back to the admin console I’m able to change the values again and the blog is still alive. Success!

Conclusion

Setting up a domain proved easier than I expected. However, fixing the blog after my mistake was a good challenge. After all the problems I ran into I can confidently say I’m able to set up WordPress blog from scratch and do some basic debugging on a kubernetes cluster. I think knowing how to mount a persistent disk and how connect to a pod are useful basic skills in kubernetes.

What’s next

For the next mini-project, I might set up a certificate for the site provided by let’s encrypt to enable https. Also, I’m thinking about setting up a monitoring and/or log solution other than kubernetes-ui dashboard to keep learning. Or I might setup a new cluster following one of the other examples in the repo. Stay tuned!