knownSubset

A polygot programmer doing web stuff

© 2015. Nathan Dauber All rights reserved.

AWS S3 Errors - EntityTooSmall

I was recently reworking an upload process at my job to make use of the Multipart Upload service that is a part of AWS S3. Everything was working fine and I was able to send the 1 MB chunks off to S3. However, once I went to complete the multipart upload I started to receive the EntityTooSmall error back from the Ruby SDK. I wasn't entirely too clear about why it would complain since it accepted all of the chunks and only rejected things once I tried to complete the multipart upload.

The AWS docs do not do much to tell you what the hell happened if you are receiving this error. You will have to dig down to these docs to finally get some clue. Apparently AWS considers 5 MB to be the minimum size of a multipart upload that they should have to encounter (excluding the last chunk).

Bypassing the AWS 16KB user-data limit for CloudInit on CoreOS

The AWS docs for CoreOS will get you set up quickly. Using CoreOS's example shows that we can pass in our cloud config via the page like they show in the docs or by using aws-cli.

aws ec2 run-instances --image-id ami-323b195a ... --user-data file://cloud-config.yaml

The cloud-config can quickly blow out to being a large file if you need to write out a bunch of files.

#cloud-config

coreos:
  etcd:
    # generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3
    # specify the intial size of your cluster with ?size=X
    discovery: https://discovery.etcd.io/<token>
    # multi-region and multi-cloud deployments need to use $public_ipv4
    addr: $private_ipv4:4001
    peer-addr: $private_ipv4:7001
  units:
    - name: etcd.service
      command: start
    - name: fleet.service
      command: start
    - name: rotate-logs.service
      content: |
        [Unit]
        Description=Rotate api logs at 5:55 AM UTC

        [Timer]
        OnCalendar=05:55:00
        Unit=start-rotate-api-logs.service

        [X-Fleet]
        Global=true

write_files:
  - path: /etc/ssl/certs/site.com.key
    permissions: 0440
    owner: root
    content: |
      -----BEGIN RSA PRIVATE KEY-----
      bloggy blog blog
      -----END RSA PRIVATE KEY-----
  - path: /etc/ssl/certs/ca.crt
    permissions: 0440
    owner: root
    content: |
      -----BEGIN CERTIFICATE-----
      bloggy blog blog
      -----END CERTIFICATE-----
      -----BEGIN CERTIFICATE-----
      bloggy blog blog
      -----END CERTIFICATE-----
      -----BEGIN CERTIFICATE-----
      bloggy blog blog
      -----END CERTIFICATE-----
  - path: /etc/ssl/certs/site.com.crt
    permissions: 0440
    owner: root
    content: |
      -----BEGIN CERTIFICATE-----
      bloggy blog blog
      -----END CERTIFICATE-----
  - path: /etc/ssl/certs/site.com.pem
    permissions: 0440
    owner: root
    content: |
      -----BEGIN RSA PRIVATE KEY-----
      bloggy blog blog
      -----END RSA PRIVATE KEY-----
      -----BEGIN CERTIFICATE-----
      bloggy blog blog
      -----END CERTIFICATE-----

However, when trying to boot an EC2 instance with a complicated cloud-config you will probably run into a limitation within AWS of the user-data being limited to 16KB. The documents seem to be wrong as I was unable to send up user-data in any format that was larger than 16KB. Once you hit this wall all AWS will tell you is:

A client error (InvalidParameterValue) occurred when calling the RunInstances operation: User data is limited to 16384 bytes

Now you can log into the instance and manually setup the machine or use some other system setup tool, but that is extra work that I would rather have coreos-cloudinit accomplish for me. To get around the limitation, I set up a systemd service to run that will download and run the real cloud-config via coreos-cloudinit.

#cloud-config

coreos:
  units:
    - name: bypass-user-data-limit.service
      command: start
      content: |
        [Unit]
        Description=Update the machine using our own cloud config as AWS user-data sucks

        [Service]
        EnvironmentFile=/etc/environment
        ExecStart=/usr/bin/coreos-cloudinit --from-url https://gist.github.com/KnownSubset/0b4569f401879b8c29df/raw/921e5b5f7882202ee5cf56e4875576f9409db6fd/default-cloud-config.yaml
        RemainAfterExit=yes
        Type=oneshot

You need the environment file to be loaded as it will supply the public and private ip addresses that coreos-cloudinit uses to do the substitutions for $public_ipv4 and $private_ipv4.

Dynamic validations via Ember-Validations

Ember-Validations is a great library to allow for validations. A small problem with it is that it expects the validations to be static. This is usually not a problem if you are making some standard web forms where the content is always the same. However in some of my internal projects the content of the form can be quite dynamic. To resolve this issue, we store the validations to run against something inside our database and load it via Ember Data.

If you look at the internals of the validator mixin you will see that the everything for validation is only set up during the init method for the mixin. So how do you solve the problem of having dynamic validations? Once you know what validations are required you can create the object that will be validated.

var dynamicValidations = this.get('validations'); //can be from anywhere (model, property, controller)
var container = this.get('container'); //needed by Ember Validations to resolve validators
var validations = {obj: dynamicValidations};
var properties = {validations: validations, obj: null, container: container};
var validateableObject = Ember.Object.createWithMixins(EmberValidations.Mixin, properties);

You now have a validateable object that you can use/reference from your templates. The bad part is that you will have to manually set the property you are validating against on the validatable object whenever there is a change.

Why are we creating unusable components?

In several spots in my Ember.js applications I have ran into a need for using a select box. Since we no longer live in the 90's I rather have my users have a rich experience and the built-in input-helpers were not cutting it.

Having ran across the chosen select box in the past I wanted a similar experience for my users. This lead me to try and integrate Addepar's Ember Widgets, but yet it just felt too bulky and I hate the idea that they weren't completely Ember components. So I attempted to roll my own, and have it so every piece of the select box would be a Ember.js component.

In my attempts to make the component I have an epiphany around customization of components. We should be able to customize without having to mess too much with component. We should never have to extend the component to make the changes required. So let us make our components allow the users define how the component works outside of the core functionality like in the following example.

A component like this allows for defining custom behavior for hover, drag, popover, clicks, and anything else since the display elements for the content would be their own components. With the ability to use components in this manner we can create a set of components with pre-defined core functionality like a grid, table, tree, and many more that anyone can use within their projects while still adding whatever additional functionality is required.Luckily this is all be possible by making use of layout property. We can do this by setting the layout during the init of the component.

Once we have it working, we can do something like this prototype which showcases multiple stylings of the same component.

JS Bin Example

We should abadon or extend any component that does not support custom usage in favor of ones that we can use beyond a single project. I would like to see a community effort grow into producing a library of components and examples in the fashion of Bootstrap for Ember.