Tuesday, March 21, 2017 around 6:30 pm
Just got home from work to find my brand new blog very slow. Which I find odd since I’m still at 18.104.22.168 and only shared the IP with a friend. So I’m jumped on the cloud shell to have a looks at the pods logs:
<span style="font-weight: 400;">$kubectl logs wordpress-3772071710-cn374</span>
...10.1xx.xxx.xxx- - [21/Mar/2017:22:44:56 +0000] "GET http://weibo.cn/5179282824 HTTP/1.1" 301 332 "http://weibo.cn/5179282824" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko" 10.92.0.1 - - [21/Mar/2017:22:44:56 +0000] "POST http://prostokvashino.ru:80/ HTTP/1.1" 404 10516 "http://prostokvashino.ru/" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9a3pre) Gecko/20070330" 1xx.xxx.xxx - - [21/Mar/2017:22:44:56 +0000] "GET http://weibo.cn/5179282824 HTTP/1.1" 301 332 "http://weibo.cn/5179282824" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 QIHU 360SE" 10.92.0.1 - - [21/Mar/2017:22:44:56 +0000] "GET http://weibo.cn/5179282824 HTTP/1.1" 301 332 "http://weibo.cn/5179282824" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)"...
A bunch of GET requests clogging down my kubernetes cluster. I have no idea why this is happening. Have I reached enough fame for a ddos attack in less than 24hrs? Is it an infinite loop somewhere in the stack(wordpress,mysql,apache,docker,kubernetes)? The bad news is I have no idea the origin. The good news is that I now have a good reason to write a blog on my blog. Everything still very meta.
Stop the Bleed
So where to start? After the logs I decided to stop the service which exposes the external IP:
$kubectl delete service wordpress
Blog down for the time being. Oddly enough after stopping the service I kept getting the same logs. Maybe an error in the stack? Next I decided to stop the wordpress pod:
<span style="font-weight: 400;">$kubectl delete pod wordpress-3772071710-cn374</span>
After a few minutes it was restarted by the cluster but I get no logs. Similarly, I tried to get logs for the MySQL pod but nothing. I used
<span style="font-weight: 400;">$kubectl describe pod wordpress-3772071710-scd0q </span>
to get the pod IP and I noticed in the events that the persistent volume failed to mount:
Warning FailedMount Failed to attach volume "wordpress-pv-2" on node "gke-k0-default-pool-1b3db445-lzl6" with: googleapi: Error 400: The disk resource 'projects/blog-162023/zones/us-east1-b/disks/wordpress-2' is already being used by 'projects/blog-162023/zones/us-east1-b/instances/gke-k0-default-pool-1b3db445-8fph'
I also tried curling the pod and got the process just hanged. Looking at the logs I see:
$ kubectl logs wordpress-3772071710-scd0qAH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.92.2.4. Set the 'ServerName' directive globally to suppress this messageAH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.92.2.4. Set the 'ServerName' directive globally to suppress this message[Tue Mar 21 22:50:16.686623 2017] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.10 (Debian) PHP/5.6.28 configured -- resuming normal operations[Tue Mar 21 22:50:16.686883 2017] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
Restarting and Updating wordpress
I’ve recreated the service again by doing:
<span style="font-weight: 400;">$kubectl create -f $KUBE_REPO/examples/mysql-wordpress-pd/wordpress-deployment.yaml</span>
Only the service gets created since the pods is already up and running.
Now I wish to update the image to wordpress:4.7.3-apache from wordpress:4.6.1-apache to . according to the guide :
<span style="font-weight: 400;">kubectl set image deployment/wordpress wordpress=wordpress:4.7.3-apache</span>
Up and running. Yet the site appears to be compromised again… delete the service for the time being.
Redeploying from Scratch
I did some research to figure out what was happening but had no luck. I even asked on the WordPress forum to no avail. I’ve decided to recreate the cluster from scratch and try again.
To no surprise I ran into issues trying to use MySQL 5.7:
2017-04-15T21:11:54.033297Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.2017-04-15T21:11:54.033464Z 0 [ERROR] Aborting
Fortunately, rolling back to 5.6 I had no issues deploying the DB and using the latest version of WordPress, that is 4.7.3.
I’ve got to admit that fixing the blog became more troublesome than expected. I even thought of giving up on the kubernetes/docker platform for the sake of simplicity but that would beat the purpose of learning how to deploy something on production for the world to see.