Categories
Amazon Web Services Cloud Architecting Uncategorized

Monitoring Disk Space with CloudWatch

I recently had a request to monitor disk space. Being that I don’t use a traditional monitoring platform but rather send all of my alerting to Slack I wondered how this would work.

There is not a direct metric in CloudWatch so we will utilize the scripts available in this guide.

You can follow along on the Amazon guide or follow some simple steps here geared towards Ubuntu based Linux distributions.

First, let’s install some dependancies:

sudo apt-get install unzip

sudo apt-get install libwww-perl libdatetime-perl

Next, we will download the scripts from Amazon:

curl https://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.2.zip -O

Once downloaded you can unpack the ZIP file:

unzip CloudWatchMonitoringScripts-1.2.2.zip && \ rm CloudWatchMonitoringScripts-1.2.2.zip && \ cd aws-scripts-mon

This will put the scripts into a directory called aws-scripts-mon inside of whatever directory you are currently in. I recommend doing this inside of /home/your-user.

There are a few ways to allow your scripts to have permissions to CloudWatch. I preferred to create the awscreds.conf method but you can also give your instance an IAM role or specify the credentials inline. If you are unsure of how to create IAM policies or roles feel free to message me and we can chat more about that.

Inside the directory there is a template file that you can utilize to generate your awscreds.conf file.

cp awscreds.template awscreds.conf && vi awscreds.conf

Modify the file as needed and save and close it.

Now let’s test the scripts to ensure functionality:

./mon-put-instance-data.pl --disk-space-util --disk-path=/ --verify --verbose

You should see a “Completed Successfully” message. If not, troubleshoot as needed.

The scripts have a lot of functionality but we are specifically looking at disk usage. I added the following line as a Cron Job:

0 * * * * /home/ubuntu/aws-scripts-mon/mon-put-instance-data.pl --disk-space-util --disk-path=/

This runs the script every hour on the hour and reports the data to CloudWatch.

Now that our data is being put into CloudWatch we need to alert on any issues. For the purpose of testing I created an alarm that was below my threshold so I could verify the alerting worked. You can adjust as you need to.

Login to your AWS Management Console and navigate to the CloudWatch Console. Your data will be placed into the “Metrics” tab. Once the Metrics tab is open you will see a section called “Linux Systems”. Navigate to this and you should metrics called “Filesystem, InstanceId, Mountpath”. This is where your metrics live. You can navigate around here and view your metrics in the graphing utility. Once you have verified that the data is accurate you can create an alarm based on this metric.

Navigate to the Alarms section of CloudWatch. Click “Create alarm” in the top right corner. Follow the steps to create your Alarm. For Metric navigate to the metric we found in the previous step. For Conditions, I chose the following:

Threshold Type: Static
Whenever DiskSpaceUtilization is…: Greater than the threshold
Than…: 45% (Note this value will change based on your actually usage. For testing I recommend setting this to a value lower than your actual usage percentage so that your alarm will fire.)

Click Next to continue. On the following page you can setup your notifications. I covered creating an AWS Chatbot here I have all of my CloudWatch Alarms sent to an SNS topic called aws-alerts. You can create something similar and have your AWS Chatbot monitor that topic as well. Once the alarm fires you should be getting an alert in your specified Slack Channel that looks something like this:

Once your alarm is firing you can fine tune your thresholds to notify you as you need!

Categories
Linux Python

Amazon S3 Backup from FreeNAS

I was chatting with my Dad about storage for his documents. He mentioned wanting to store them on my home NAS. I chuckled and stated that I would just push them up to the cloud because it would be cheaper and more reliable. When I got home that day I thought to myself how I would actually complete this task.

There are plenty of obvious tools to accomplish offsite backup. I want to push all of my home videos and pictures to an S3 bucket in my AWS environment. I could:

  1. Mount the S3 bucket using the drivers provided by AWS and then RSYNC the data across on a cron job.
  2. Utilize a FreeNAS plugin to drive the backup
  3. Build my own custom solution to the problem and re-invent the wheel!

It is clear the choice is going to be 3.

With the help of the Internet and I put together a simple Python script that will backup my data. I can then run this on a cron job to upload the files periodically. OR! I could Dockerize the script and then run it as a container! Queue more overkill.

The result is something complicated for a simple backup task. But I like it and it works for my environments. One of the most important things is that I can point the script at one directory that houses many Symlinks to other directories so I only have to manage one backup point.

Take a look at the GitHub link below and let me know your thoughts!

[GitHub]

Categories
Amazon Web Services

Setting Up AWS Chatbot

Amazon Web Services pushed their new Chatbot into beta recently. This simple bot will allow you to get alerts and notifications sent to either Slack or Amazon Chime. Because I use Slack for alerting I thought this would be a great tool. Previously I utilized Marbot to accommodate a similar function. Marbot is a great product for teams as it allows a user to acknowledge or pass an incident. I am a team of one so this feature is nice but ultimately not useful for me at this time.

Let’s get started!

Navigate to the new AWS Chatbot in the console

On the right-hand side click the drop-down menu to choose your chat client. I am going to choose Slack because that is what I use. I assume the process would be the same for Chime. You will be prompted by Slack to authorize the application. Go ahead and hit “Install”.

On the next screen, we will get to our configuration options. The first being to choose our Slack Channel:

I chose the public channel that I already have created for Marbot #aws-alerts. You can do what you want here. Maybe you want a private channel so only you can see alerts for your development environment!

The next section is IAM Permissions

I chose to create an IAM role using a template and utilized the predefined template and just made up a role name called “aws-chatbot-alerts”.

The last configuration options is for SNS topics

You can have your bot subscribe to SNS channels to receive notifications to publish there as well. I don’t currently use any so I skipped this section but, this could be super useful in the future! Look for future posts about this idea!

I will update this post soon with how to create the chatbot using the CLI and/or CloudFormation

Categories
Amazon Web Services

Copying Files To & From an AWS S3 Bucket

Recently I needed to download an entire bucket worth of data for an offsite backup. Easy right? Go to the Amazon Web Services Console and hit download! WRONG.

You can download individual files but not an entire bucket. Seems silly. Luckily there is an easy way to do it via the Amazon Web Services CLI. Enter simple commands:

$ aws s3 cp s3://YOUR_BUCKET/ /LOCAL_DIRECTORY --recursive

Let’s dissect this just a little bit. The first couple of options in the command should be pretty self-explanatory. We are going to use the AWS CLI, we chose S3 as our service and then the ‘cp’ means we are going to copy. Now, there are a bunch of other options that you can do here. I suggest taking a look at the documentation here to learn more. After that, you simply add in your bucket name, note the trailing forward slash, then where you want to put your files on your local machine. Finally, I added the --recursive flag so that it would read through all the lower directories.

Ultimately a very simple solution to transfer some data quickly! The AWS S3 CLI functions very similarly to that of your standard directory functions. So, feel free to poke around and see what it can do!

Categories
Amazon Web Services Cloud Architecting

AWS CLI For CPU Credit Balance

Here is how you create a CloudWatch alarm to monitor CPU Credit Balances less than a certain amount:

aws cloudwatch put-metric-alarm --alarm-name YOUR NAME HERE--alarm-description "Alarm when CPU Credits is below 200" --metric-name CPUCreditBalance --namespace AWS/EC2 --statistic Average --period 300 --threshold 200 --comparison-operator LessThanThreshold --dimensions Name=InstanceId,Value=INSTANCEIDHERE --evaluation-periods 2 --alarm-actions ARN:YOURSNSTOPIC

CloudFormation Template:
https://github.com/avansledright/CloudFormation-CPU-CREDIT-BALANCE

Categories
Linux Technology

Lessons Learned from Migrating 17TB of Data

I finally pulled the trigger on some new hard drives for my home NAS. I am migrating from a 5U Server down two a small desktop size NAS. Ultimately this removes the need for my 42U standing rack.

I did this transfer a year or so ago when I did a full rebuild of my server but forgot to take any notes on the processes that I used. Instant regret. I remembered utilizing Rsync to do the actual transfer and I assumed that I mounted both the existing NAS to an NFS share and the new NAS through NFS. Both these mounts would reside inside a throwaway virtual machine on my application server.

I used the following Rsync command to start.

rsync --ignore-existing -ahzrvvv --progress {Source} {Destination}

To break this down a little bit:

–ignore-existing: This will ignore any existing files that copy over

-a: Archive flag. This preserves my data structure

-h: Human readable. If this flag exists for a command, use it. It makes things much easier to use.

-z: Compression. There are a bunch of different compression options for Rsync. This one does enough for me.

-r: This makes Rsync copy files recursively through the directories

-vvv: I put triple verbose on because I was having so many issues.

–progress: This will show the number of files and the progress of the file that is currently being copied. Especially useful when copying large files.

Now, my command changed over time but ultimately this is what I ended on. My source and destination were set to the respective NFS mounts and I hit [enter] to start the transfer. I left it running on the console of my Virtual Machine and walked away after I saw a handful of successful transfers. Assuming everything was going fine I went about my day as 17TB is going to take a while.

A few hours later I decided to check in on my transfer and saw that it had gotten stuck on a file after only 37KB of data transfer! Frustrated, I restarted the process. Only to see the same results later on.

After updating, downgrading, and modifying my command structure I came to the realization that there must be an issue with transferring between to NFS shares.

I am still researching why this happens but to me, it seems as though when the transfer starts the files are brought into a buffer somewhere within the Linux filesystem which gets maxed out causing the file transfer to stall. Almost as if the buffer can’t send the new files fast enough.

When I switched the transfer to utilize SSH instead of NFS to NFS the transfer completed successfully.

If someone has some information regarding how this works I would love to learn more.

Categories
Amazon Web Services Cloud Architecting Technology

Encrypt an Existing EBS Volume

Say you have an existing EBS volume on Amazon Web Services that you wanted to encrypt. How would you do that? The following guide shows you how to do so via the AWS Management Console.

  1. Login to your console.
  2. Navigate to the the EBS Volume you would like to encrypt

3. Right click on your colume and create a snapshot.

4. I always give my snapshots descriptions. But we are going to end up deleting this one.

5. Make a copy of the snapshot you created in step 4.

6. In the copy settings you simply need to choose to encrypt the volume. You can specify the encryption keys to use. For this guide we will just use the standard EBS encryption key.

Once you have your new encrypted snapshot you can easily create a volume from that snapshot and then re-attach it to your instance!

Categories
Technology

Fixing Unadoptable Unifi Devices

I wrote an article about this before that utilizes Robo3T. I figured I should also have a version for those of you who utilize SSH and Command Line.

DISCLAIMER: I am not responsible if you break anything. If you need help let me know before you create a big mess!

EDIT: I wrote a Python Script that can handle all of this for you just enter in your MAC address. Grab it here: https://github.com/avansledright/unifideletedevice

SSH into your Unifi Controller utilizing whatever means you typically use.

Connect to MongoDB by issuing the command:
mongo --port 27117

If you are utilizing a different port then change the port flag.

Once connected select the Unifi Database:

use ace

Then you can utilize the following queries to preform actions:

Find device:
db.device.find({ 'mac' : 'XX:XX:XX:XX:XX:XX' })
Remove device:
db.device.remove({ 'mac' : 'XX:XX:XX:XX:XX:XX' })

Should you want to find what site a device is registered to you can utilize the “Find Device” query from above. In the JSON output locate the Site ID. Then utilize the query below and replace the X’s with your found site ID. The result should be a nice JSON output with the name of the site.

Find site query:
db.site.find(ObjectId('XXXXXX'))

Categories
Technology

Dialpad – A Review

Dialpad is an online voice over IP phone system focused on being the simplest phone system you have ever used. Is it? So far I sure think so.

Full disclosure: One of the businesses that I am employed by sells phone systems. It isn’t Dialpad.

When you run a business people inevitably want to call you. For the longest time I avoided having a phone number or giving out my cell phone number. I just wanted to avoid phone calls all together. Life is so much easier over email. But eventually, you need to move on and be able to accept a phone call.

I started using Twilio. Twilio was born in the cloud, runs on Amazon Web Services seems right up my alley! But it is not a phone system. It is very basic unless you want to spend hours programming on another system to get it to do what you want it to. I didn’t have time for that. But, it did allow me to have a phone number, forward calls, and forward text messages to my existing cell phone. Good enough for now.

My business is growing though. I need more features. With Twilio I still have to respond with my personal cell phone number. This is not great for a number of reasons. Most notably, I don’t want to give out my personal number anymore! This is where Dialpad comes in. Upon sign up I received a new business phone number, a personal phone number AND a conference line.

So I modified my existing Twilio number to forward to my new business line. You can port numbers into Dialpad if you pay for a more advanced plan. As I am unsure if I will stay with this software I opted to leave my number at Twilio. I then added myself as a forwarding user so that calls can come into my cell phone if I am away from my desk. All of this is done through a very user friendly web interface. You can also link it up to your GSuite accout to automatically add new users to Dialpad and put them into their respective call group.

After all of this was setup, I recorded some greetings and downloaded the desktop app. It works exactly as you would expect it to without any issues. The mobile app functions quite well. It has some quirks to it on the messaging side but overall it does what I need it to do.

One of the most interesting aspects of Dialpad is their Voice AI feature. While you are on a call it can live transcribe the call for you in the desktop app. Once the call is over it will analyze it and give you feedback. Just so happens the call I was on was with a client who was unhappy with the way their sales were going for the year so it flagged the call for lots of “Negative sentiments”. This is a very interesting feature that I will be keeping tabs on going forward.

Overall: If you want an easy to setup, full featured phone system with a decent price and don’t care about having a physical deskphone, Dialpad is a great option!

Categories
Technology

Counting Web Requests

I manage a ton of web servers. Occasionally I see attempts at flooding the servers with traffic. Typically in a malicious way. Generally these are just small attacks and nothing to write home about. But, I wanted a way see how many times a server was getting a request from a specific IP address.

Obviously this would be very challenging to accomplish by just looking at the logs. So, I put together a small Linux command that will read and count Apache requests based on unique IP addresses.

cat access.* | awk ‘{ print $1 }’ | sort | uniq -c | sort -n

Try it out and let me know what you think!