Using AWS Lambda for Web Video Transcoding

Posted on Thu 03 September 2015 in aws • Tagged with lambda, elastic-transcoder, video

Often your creative team will produce master videos in 4k or 1080p, but you need to downcode these videos into 720p/1080p for web broadcasting. Here we automate transcoding of masters into web-friendly formats like 720p h264 mp4 & webm.

AWS Elastic Transcoder is a cloud video transcoding service. At it's simplest it transcodes video files from one bitrate, framerate, codec, container, etc--into another. By default you trigger new jobs either manually in the aws console or via the rest API. And naturally all inputs & outputs are saved in S3.

Transcoder setup includes creating a pipeline and presets. Then for each …

Continue reading

Wordpress Cron on Opsworks

Posted on Thu 27 August 2015 in aws • Tagged with opsworks, chef

By default Wordpress uses it's own pseudo-cron which triggers with every request. Obviously this is wasteful since (a) the queue needs to be inspected with every GET and (b) jobs like publishing articles will interfere with serving content.

Some suggest calling the wp-cron.php GET request with curl in a cron like this

* * * * * curl http://www.mysite.com/wp-cron.php

but that's sub-optimal since it needlessly ties up a worker during the cron execution.

If you're using chef or Opsworks, here's a tidy way to install the system cron to execute without interfering with your webserver.

First, disable the Wordpress …

Continue reading

HTTP Redirects with Cloudfront & S3

Posted on Tue 18 August 2015 in aws • Tagged with aws, cloudfront, s3, http

Redirects can account for a significant share of direct traffic so taking a few minutes to optimize them is worthwhile.

Using Cloudfront & S3 for redirects will improve responsiveness, reduce server load and improve management (since they are managed via aws-cli or the console).

Let's say you have a typical .htaccess redirect like this.

RewriteEngine On
### re-direct to www
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

Or worse, it could look like this in your index.php

$protocol = (@$_SERVER["HTTPS"] == "on") ? "https://" : "http://";

if (substr($_SERVER['HTTP_HOST'], 0, 4) !== 'www.') {
    header('Location: '.$protocol.'www.'.$_SERVER …
Continue reading

Better Battery Statistics with Battery Historian

Posted on Mon 17 August 2015 in android • Tagged with android, battery, debug

Continuing the relentless quest to keep my phone speedy, I stumbled upon a developer tool that is useful to anyone needing to troubleshoot a slow, short-living or overheating phone -- Battery Historian

battery historian

Battery Historian shows you a much more detailed and informative battery stats chart, highlighting the individual apps and sync services which are keeping your phone awake/busy in the background. It also shows network, wifi status, gps and more.

Using this tool I identified that the Facebook Messenger app was waking up to send stats. Also, Google+ was syncing for ~40s in the bakground at times.

tl;dr

$ git …
Continue reading

Debugging Android Performance & Battery Issues--Like a Developer

Posted on Sat 15 August 2015 in android • Tagged with android, battery, debug

I have a frustrating relationship with my phone's performance. I can cleanup my phone for a few days, but it tends to revert to being sluggish within no time. I've had dozens of devices and they all suffer from this.

There's a lot of voodoo about Android Performance and Battery life--task managers, factory resets, etc.

Here's a more developer-oriented process using adb .

Using ADB to identify process hogs

By connecting your device to the Android SDK, you can use ADB to identify process hogs. If you can, just remove the app. Otherwise, delete it's data (see pm clear below)

$ adb …
Continue reading

Validating side-loaded APKs

Posted on Thu 13 August 2015 in android • Tagged with android, apk, debug

I was desperate to try Hangouts 4.0 for Android, but suspicious of side-loading. I wanted to verify the APK signature cert had Google's fingerprint of

38:91:8A:45:3D:07:19:93:54:F8:B1:9A:F0:5E:C6:56:2C:ED:57:88

Here's how to check the signatures on an APK, as usual, in shell functions (JDK needed)

apk-check () {
    jarsigner -verify -verbose -certs $1
}

apk-print-cert () {
    keytool -list -printcert -jarfile $1
}

# usage
# make sure it's verified
$ apk-check *apk|grep verified
  s = signature was verified
jar verified.
# show …
Continue reading

Opsworks -- Quickly Listing Hosts on the Command Line

Posted on Tue 11 August 2015 in aws • Tagged with aws, opsworks, cli

Here's a great example of using the aws-cli to speed up your life. Uses jq and aws-cli

  # bash / zsh function
  function opsworks-hosts-prod () {
    aws opsworks describe-instances --stack-id=fffff-fffff-ffff-fff-fffffff | jq '.Instances[].PublicDns' | grep -v null | sed s/\"//g
  }
  # usage
  $ opsworks-hosts-prod
  XXXXX.compute-1.amazonaws.com
  XXXXX.compute-1.amazonaws.com
  XXXXX.compute-1.amazonaws.com
  XXXXX.compute-1.amazonaws.com
Continue reading

First Things First, on AWS

Posted on Fri 07 August 2015 in android • Tagged with aws, secuity, tutorial

I was chatting with a buddy who was moving his web sites from dedicated hosting to AWS. Let's just say the FTUE isn't great. That triggered a quick brain-dump of what you should do when you first get started with AWS.

  • understand pets v cattle. In aws all resources should be "cattle", not pets. Periodically terminate instances to test this.
  • activate cloudtrail (in all regions). Use Loggly to index cloudtrail (free or ~$20/mo)
  • create restricted IAM users. Never use your root acct. Activate MFA.
  • Use IAM ec2-instance roles instead of stored credentials whenever possible.
  • Get familiar with IAM management …
Continue reading

On Software Scaffolding

Posted on Thu 09 July 2015 in aws • Tagged with monitoring, software

waterloo_bridge_1815 A new lightrail line is being built in my city with bridges passing over the major boulevards.  Seeing the elaborate scaffolding evoked comparisons to software engineering.  What does scaffolding look like in software? Does software need to be erected like a bridge via scaffolding?  Without a doubt: yes.

Here are some elements of software “scaffolding”:

  • Error log instrumentation with a formal error log schema (i.e. errors are uniquely identifiable in a MECE schema)
  • Operational instrumentation with reports , dashboards and alerts
  • Performance profiling on methods, database calls, rest calls, system calls and any blocking IO.
  • Client-side performance instrumentation and sampling …
Continue reading

Opsworks before-migrate.rb

Posted on Tue 09 June 2015 in aws • Tagged with chef, opsworks, aws, devops

Opsworks is a convenient, powerful and free service provided by AWS to simplify the management of EC2 nodes.  The real power of the system is exposed through customizing various stages of the instance lifecycle by creating custom-tailored chef-solo recipes.

While Amazon provides a powerful deployment layer for PHP applications, it stops short once the PHP code has been checked out of git.  For Laravel or other composer apps, you’ll have to customize your deployment.  The most elegant and straightforward method is through custom deployment hooks.  Here’s how to build a before_migration.rb script to build a Laravel app …

Continue reading