They are wildly useful for membership checks without loading the entire set into memory in cases where occasional false positives are acceptable. This can make certain cases of checks far more reasonable where scanning the entire set every time would be prohibitively expensive.

Whenever I try to explain them to someone, they tend to get confused. To that end I wanted to put together an oversimplified explanation I can point friends and coworkers to on how to think about these sorts of filters when these questions arise.

This will not fully explain how probabilistic filters work on a technical level. It will arm you with a simplified way to think about them. A working mental model.

Definition

The general idea of a probabilistic set-membership filter is that it is a memory-efficient way to check membership in a set. An item once added to the filter is guaranteed to always pass the filter. An item never added to the filter will probably fail the filter but may sometimes pass.

Example

A good example of probabilistic filtering in action is Google Safe Browsing. Instead of sending your browser an enormous list of 5-billion+ known dangerous URLs, Google sends a compact probabilistic representation of that list. If a URL fails the filter, it is not in Google's list of known dangerous URLs. If it passes the filter, the browser verifies it with Google against their full database to eliminate false positives.

The Mental Model

Let's start by imagining your data. Let's let the following shapes represent the individual items in our set.

And let's let this "wall" represent our filter.

Now imagine forcing our data shapes through the wall and the impression they would make. This can be imagined as adding items to our filter. There is now a hole in the wall in the shape of our data. As we continue pushing all our items through, we are additively making the hole larger.

Imagine the wall is now locked from further changes. It becomes a filter that only certain shapes may pass through.

One can see how all three of our shapes in our set would successfully "pass" the filter, while some shapes not in our set would fail to pass the filter.

One can also imagine shapes we did not define that would pass. This is how false positives occur when using filters like these.

Digitization

So that's all fine and good for a wall in the physical world, but now we want to digitize this. Let's imagine our wall as a bitmap - a literal map of bits. A zero where our wall has been untouched, and a one wherever it was pierced.

To check if an object passes our filter, we simply check if its pattern fits into our filter's matrix.

Here we can see if we digitize our diamond from earlier, its enabled bits fit cleanly into the marked bits of our filter, and thus it passes the filter.

In many real implementations the "shape" is produced by hashing the item several times, which selects positions in the bitmap to set. The bitmap would also be magnitudes larger than the one in our illustration.

One can easily imagine how this could save a lot of memory. Instead of passing around an entire set with potentially millions or billions of members, you only need to pass the serialized bits of the filter.

As I said at the beginning, this is not exactly how they work. This is a drastic oversimplification, and the exact inner workings vary by filter type and configuration. This is just a mental model that can be helpful when thinking about them.

TL;DR: To fully implement the PSR-16 specification as defined, an implementation MUST be contravariant of the defined interface in its accepted parameters when implementing v2 or later.

I work on an application with a proprietary caching layer. Recently I wanted to build a PSR-16 "Simple Cache" adapter for it, and started going through the spec and provided interface in detail. I wanted my implementation to be as correct as possible.

While working my way through getMultiple/setMultiple/deleteMultiple I noted something that seemed like an oversight.

/**
 * Obtains multiple cache items by their unique keys.
 *
 * @param iterable<string> $keys    A list of keys that can be obtained in a single operation.
 * @param mixed            $default Default value to return for keys that do not exist.
 *
 * @return iterable<string, mixed> A list of key => value pairs. Cache keys that do not exist or are stale will have $default as value.
 *
 * @throws \Psr\SimpleCache\InvalidArgumentException
 *   MUST be thrown if $keys is neither an array nor a Traversable,
 *   or if any of the $keys are not a legal value.
 */
public function getMultiple(iterable $keys, mixed $default = null): iterable;

Take a moment to read this carefully

@throws \Psr\SimpleCache\InvalidArgumentException
  MUST be thrown if $keys is neither an array nor a Traversable
  or if any of the $keys are not a legal value.

Then, inspect the definition

public function getMultiple(iterable $keys, mixed $default = null): iterable;

The takeaway here should be:

iterable $keys

To implement getMultiple, we MUST throw a \Psr\SimpleCache\InvalidArgumentException when $keys is invalid, yet $keys is defined as iterable which makes it impossible to handle if an implementation matches this signature.

Passing anything non-iterable will throw a TypeError before the code even runs. This would make it impossible to throw an InvalidArgumentException.

Parameter types were added to the interface with the release of v2. My suspicion was that the @throws had simply not been updated, because if the parameter type is explicitly defined as iterable, the function cannot throw a custom exception.

I originally suspected this would make it impossible to fulfill the spec. However, I had not considered contravariance. The implementing methods accepted parameter types may be wider than the interface it is fulfilling. In this instance, $keys does not need to be defined as iterable in the implementation.

This means a technically complete implementation must omit iterable and look something like

public function getMultiple($keys, mixed $default = null): iterable {
    if(!is_array($keys) && !$var instanceof Traversable) {
        // Must implement \Psr\SimpleCache\InvalidArgumentException;
        throw new ExampleException;
    }
}

Note \Psr\SimpleCache\InvalidArgumentException is an interface. An implementation must define a concrete exception class that implements it before it can be thrown

So while it's not impossible to implement, it certainly is strange to implement. One can only correctly implement the spec by manually checking $keys type like this.

Doing a little investigation, it seems like most major implementations still target v1 || v2 || v3 and as v1 did not have typed parameters, neither do the implementations iterable parameters. This means most major implementations are implemented correctly.

However, there are at least a couple of implementations that get this wrong.

• laminas/laminas-cache
• yiisoft/cache
• phpfastcache/phpfastcache

I'm sure there are others, but I found these three in my quick search. It is a very easy mistake to make when not implementing v1.

This is not Java. There are no checked exceptions forcing anyone's hand. In practice, this is a minor quirk of the spec. If a TypeError gets thrown instead of an \Psr\SimpleCache\InvalidArgumentException, almost certainly nothing meaningful breaks. I doubt anything depends on this behavior. It’s just an inconsistency between what the spec says and what implementations are doing. It is nothing to lose sleep over, but it might be worth fixing for consistency's sake.

In summary, to fully implement the getMultiple/setMultiple/deleteMultiple methods of the PSR-16 spec, one cannot simply copy their definitions directly from the CacheInterface. The implementation must check the validity of their iterable parameters and not have them defined as an iterable.

By default, if you try to run an unsigned binary in Terminal, you will see an error like:

Apple could not verify "[binary]" is free of malware that may harm your Mac or compromise your privacy

Instead of offering any help, macOS simply offers to move the binary to the trash. Very frustrating.

You find a lot of complicated solutions online, such as manually unquarantining and self-signing the individual binaries with:

$ xattr -dr com.apple.quarantine ./[binary]
$ codesign -s - --deep --force ./[binary]

Fortunately, there's a simple solution: add Terminal as a system Developer Tool in System Settings.

This will just allow it to run unsigned binaries. No fuss, no muss.

To do this:

1. Open System Settings
2. Search for "developer"
3. Click Allow applications to use developer tools in the sidebar.

If Terminal is not already listed, click the + and search for it:

Search for Terminal in the file dialog that appears and select it.

Once added, ensure the toggle next to Terminal is enabled.

Finally, restart Terminal and everything should now work:

There you have it. You may now run any unsigned binary from Terminal without issues.

They've had just a minute to marinate. In some parts of the community they've been an unpopular addition, however I don't particularly mind them. The old alternatives seem a little more "Go" to me, but the new iterators are fine.

One thing however I am mildly disappointed about is that there isn't just a generic iterable type similar to the generic comparable. It would be nice to have an imaginary iterable type that could accept maps, slices, arrays, and iterators. As I understand it, iterators combined with slices.All and maps.All are the blessed alternative to that. It works, I'm using it, but I don't love it.

All that's neither here nor there, however, and not what I'm here to talk about.

The part that smells funny to me is that range now accepts any already existing function that happens to fulfill either of the following interfaces.

func(func(any) bool)
func(func(any,any) bool)

It would be less weird if that wasn't also a little troublesome.

Example time!

The first interface, known as Seq takes a yield method as an argument that itself accepts any single parameter. The second interface, known as Seq2 similarly takes a yield method that accepts any two parameters as essentially a key/value tuple.

Note that yield is not a magic keyword as in many languages, but a passed-in callable. This is very akin to Go allowing you to name receivers rather than magically defining this.

At first blush to the uninitiated, this iterator looks fine. It seemingly runs fine as well.

package main

import "fmt"

func mightBeAnIterator(f func(string) bool) {
    f("foo")
    f("bar")
    f("baz")
}

func main() {
    for x := range mightBeAnIterator {
        fmt.Println(x)
    }
}

foo
bar
baz

What's the problem then?

The problem is iterators MUST handle early exit of the loop manually. With the current interface fulfilling "iterator", if we just add what seems a harmless break to our loop as follows, suddenly the code working moments ago panics.

    for x := range mightBeAnIterator {
        fmt.Println(x)
        break
    }

foo
panic: runtime error: range function continued iteration after function for loop body returned false

Somewhat surprisingly, as Go exits the loop, it allows the iterator to continue executing. In doing so, however, it signals the iterator through the result of the yielding method that it should return. This is to allow for teardown. For example, imagine an iterator that opens files or a socket; continuing execution allows it to close them before returning.

However, Go will panic if the signal to return is not honored and the yield function is then invoked again by the iterator.

To prevent our example iterator from panic-ing, we simply have to listen for the signal to exit, as shown below.

func mightBeAnIterator(f func(string) bool) {
    if !f("foo") {
        return
    }
    if !f("bar") {
        return
    }
    if !f("baz") {
        return
    }
}

And when run, now we receive simply

foo

I would imagine the reason for the panic is to prevent iterators from accidentally running wild.

To my personal taste, I would have strongly preferred honoring the exit signal to be optional, and following superfluous yield calls being quietly ignored and returning false.

I can certainly imagine cases where an iterator would want to finish regardless of early exits by its consumer. While still possible, it's awkward to implement.

We are certainly in a place where, should they decide to go back on the panic behavior, it would not break any existing code. That's not a terrible place to be.

All this is to say it smells a little funny that such a simple and likely somewhat common interface has this magic range logic available to it.

Expecting the implementers to implement it correctly and panic-ing if they don't smells kind of Worse is Better to me. While one could complicate Go's iterator interface to enforce stricter guarantees around safe iteration, doing so would add unnecessary complexity and detract from Go's philosophy of simplicity.

It's one of the rare occasions where I wish interfaces had to be intentionally met. It's strange to me that anything that smells like an iterator can be passed to range, even if it doesn't implement the intended handshake.

It smells funny, but that's probably OK. It almost certainly falls into the case of "things that will never be a real problem in the real world". I look forward to the oncoming brave new world of libraries expecting iterators.

While some are naturally somewhat gifted at debugging, it's important to understand that the skill can be learned. In fact, even very experienced developers can struggle with it

I've known developers who stare at their code trying to reason their way through it. I've known developers who will change things at random until the problem goes away. I've known developers whose technique I describe as "I've tried nothing and I'm all out of ideas".

I take a very structured approach, and this is something I've meant to write down for years now. Solving 90% of the of the problems you will face comes down to three simple steps.

1. Identify Outputs
2. Identify Inputs
3. Repeat

Identify Outputs

There is a bug in your code; that much we know. It's causing something to not work as expected. The first step in debugging is to identify what that something is.

Determine what your expected output is, and what the actual output is.

Walk through the call stack to find where the output diverged from expected.

Identify Inputs

This step is crucial and often the most challenging.

In an ideal world we'd only be debug pure functional code. The only inputs would be parameters, and all calls within are deterministic.

We don't live in an ideal world.

We often find ourselves debugging code that inherits values and state from elsewhere. This can be self-fulfilling, such code is usually hardest to test and most prone to errors.

Sources of state include, but are not limited to:

• Members of the containing class or its parents
• Non-deterministic values and methods
  • The current time
  • Random number generators
  • Potentially flakey network services
  • Databases
  • File systems
  • User input
• Global variables
• Environment variables
• Configuration files

This list is far from exhaustive and meant to give you an idea of the kinds of things you might need to consider.

While debugging, it's important to identify and note all these inputs. We then need to understand their state at the time of the error. This is often where I see developers struggle. Some part of their code can vary and they have failed to identify it.

Ways to identify their state logging, or by using a debugger to inspect the state of the system at the time of the error.

It comes down to a is a matter of personal preference as well as comfort with the tooling.

I know my way around my debugger, and yet I more often than not am inclined to just spit the state to standard out. I find this to be the most effective way of debugging. I know developers who are very good at using debuggers. I also know developers who spend a lot of time trying to make sense of their debugger.

I've never met a developer who is bad at using print statements.

The most effective debugging tool is still careful thought, coupled with judiciously placed print statements.

– Brian Kernighan

Repeat

After identifying the code's outputs and inputs, use this information to track down the problem. You will need to repeat this process recursively through the call tree.

We work our way through the call stack, identifying the unexpected outputs and inputs as we go. Keep this up and we will locate the root cause.

Other Thoughts

Identifying outputs and inputs isn't limited to values of variables. It's about the state of the system as a whole - including where boundaries get crossed.

You need to understand how the system handles crossing boundaries and ensure that the handoff results as expected. Understanding how the system manages crossing these boundaries is crucial for identifying where errors might occur.

You also may need to understand the state of the larger system at the time of the error. "The system" is not limited to the code you're working on. It can also include the state of the database, the state of the file system, the state of the network, cookies, sessions, the end user's browser, etc.

When you've identified the issue, write a test that reproduces it. This confirms you've fixed it and understood it, as well as preventing its recurrence. If you don't already have a strong test suite, regression tests are a great place to start.

Conclusion

I hope this guide helps you improve your debugging skills. Debugging is a skill that you can learn and improve over time.

Debugging is not just about fixing errors. It's an opportunity to learn your codebase and how to enhance it.

And remember, careful thought, combined with well-placed print statements, remains the most effective debugging tool.

The MIT License is very simple. It basically says you can do whatever you want with the code, but you have to include the license and copyright when redistributing the code itself. It does not however require you to open source any modifications you make to the code, nor provide attribution to the non-code products of the work.

With my libraries, this has served me very well. It's great, people use and contribute fixes back, and often donate back to me. I love it. I want you to use my code in your proprietary software. I open sourced it so you could do that. I specifically build open source libraries because I solved a problem I thought other people might face and wanted to save them time. Lifting the sea for all boats and all that. I have zero problem continuing to use the MIT License on my libraries.

I also open sourced the tools I host on my website, including but not limited to Pixel Circle / Oval Generator and Batch RewriteRule Generator. The MIT License seemed like the natural choice because I've been using it for everything I've released since the early 2000's.

I'd considered these as just part of my website far more so than something I thought people would reuse. My intentions open sourcing these tools, unlike my libraries, was to promote the community helping me to improve them - and they certainly have done so. I have gotten so much great feedback and contributions to these tools, and I am very happy with that. Massive shout out to Tyson453 for his insanely awesome work improving the speed of the Pixel Circle / Oval Generator.

Against my well meaning intentions however, websites re-hosting my tools have been popping up like weeds. In some cases, they are even beating me in search results for my own tools. With noted exception, they don't credit me as the author or provide any sort of link back. Many of them have made minor or major modifications to the tools, and next to none provide the source to those modifications. A number of them even have the gall to post links advertising them in the comments of my own tools.

Most irksome of all, in a fair number of cases they sit centrally on pages covered in ads and SEO keywords. My tools are being associated with a genuinely bad user experience.

This is all perfectly legal and allowed under the MIT License. That's on me. It just is not in the spirit of what I intended. I didn't have the foresight to see this coming. I didn't think people so lacked in the spirit of open source. I wanted to promote community contributions, not to have them monetized by other people who don't even provide the source to their modifications. I wanted to grow the tools as a community, not have closed source forks of them overtake my own open source versions.

There's not a lot I can do. I could try to contact the owners of these sites and ask them to provide attribution, but there's no obligation for them to do so. I could try to contact the owners of these sites and ask them to provide the source to their modifications, but again there is no obligation for them to do so. I really wish to see the changes shared alike, and the vain part of me would like attribution as the original author. At the end of the day though I chose the MIT License and it is what it is. It simply does not require any of that.

On top of everything else, until very recently and for completely unclear reasons, my website Donat Studios was completely blocked from Bing search results. This in turn meant I didn't show up in DuckDuckGo or Yahoo results either. This meant that for a time the only results when searching these sites for my tools were these re-hosted versions.

I actually managed to reach a human at Bing and get this resolved, but it took a lot of work and I'm still not sure why it happened in the first place.

I am considering relicensing my tools under some sort of Attribution-ShareAlike license similar to the BY-SA the content on this site is licensed under. It would not apply to current and past versions of the tools as that code is licensed MIT and you can't un-ring that bell. It would however apply to future versions. This would still promote community contributions, but would also require that any modifications be shared alike meaning that the modified source would have to be provided and I would have to be credited as the original author.

At the very least I will definitely think twice about the license I choose for future tools. I don't want to be a jerk about it, but I also don't want to see them monetized by others without even a link back to my website.

You can fix the problem by replacing number with ReturnType<typeof setTimeout>

Similarly you can replace number with ReturnType<typeof setInterval> if you are seeing the problem with setInterval.

If you want to know more about it, read on.

I've run into this problem a number of times where I'm developing frontend TypeScript and after I import a package, I suddenly find throughout my project I'm receiving:

error TS2322: Type 'Timeout' is not assignable to type 'number'.

If you are working on a Node project rather than a front end project this can also manifest itself as the opposite

error TS2322: Type 'number' is not assignable to type 'Timeout'

Usually this happens because some dependency included @types/node under dependencies instead of devDependencies. When that happens, Node's global types leak into your project—even in frontend environments—causing setTimeout to return a Timeout object instead of a number.

The heart of the problem is that setTimeout() and setInterval() in Node API return a Timeout whereas in the HTML DOM Window API, they return an integer.

The ideal solution would be for the offending package to fix the problem themselves, and for you to update to the corrected version of the package. This is a hassle and might not be possible, with the issue going ignored for months if not years - as on Facebook's Jest package ^{[1] [2]}.

For example, consider the following:

let x : number = setTimeout(() => console.log("foo"), 1000);

Your immediate reaction might be to either change the definition to let x : Timeout or eliminate the explicit type definition all together à la

let x = setTimeout(() => console.log("foo"), 1000);

Both of these options work.

The first option of setting it to Timeout suffers from two issues though. Firstly it's fragile - presumably you're not explicitly including @types/node so the change to the environment is a side effect you're coding around. Should the specific packages you include change, or said package correct this it will break. Second, it's ambiguous. Timeout isn’t a global type—it's specific to Node's type system—so using it assumes a Node environment even if you're working in the browser. That can confuse collaborators or tooling down the line.

The second option of implicit typing will work in a lot of cases just fine. There are many cases where you'd want for instance to define the variable in a higher scope than you are setting its value - as a trivial example

class Foo {
    private x? : number;

    start() {
        this.x = setTimeout(() => console.log("foo"), 1000);
    }

    stop() {
        clearTimeout(this.x);
    }
}

The answer is frustratingly simple. We can just use the ReturnType utility type to set our type to the return type of setTimeout. We can do this using ReturnType<typeof setTimeout>.

Our examples from above become

let x : ReturnType<typeof setTimeout> = setTimeout(() => console.log("foo"), 1000);

and

class Foo {
    private x?: ReturnType<typeof setTimeout>;

    start() {
        this.x = setTimeout(() => console.log("foo"), 1000);
    }

    stop() {
        clearTimeout(this.x);
    }
}

Similarly, if you are using setInterval you can utilize type ReturnType<typeof setInterval> for similar purposes.

For a very long time now, I've been wanting to be able to accept files from an end user and process them with Go on their local machine. Basically I just want to read files from file inputs. I knew it had to be possible, but could not in all this time find a good walkthrough online. A couple nights ago I set my mind to it and figured it out.

Essentially we need to execute the following JavaScript, but in Go via the JS syscall API.

So after transliterating that JavaScript into Go syscall/js calls, we then using js.CopyBytesToGo to pass the given Uint8Array back to our WASM runtime where we can do with the bytes as we please.

Note, given the keyword any here, this example requires Go 1.18 - the syscall/js library is still considered experimental and expect it to break on major go releases.

This is as written for brevity and could be improved in code safety in a number of pretty obvious ways.

The iframe below runs the compiled WASM and spits out any file you upload to it, via Go.

I recommend every developer turn off automatic garbage collection in git. It's one of the first things I do setting up a new machine - and you should do so too.

For a happier life, go to your terminal right now and run:

$ git config --global gc.auto 0

I promise you won't regret it.

The full story

Turn off garbage collection? Surely, I must be crazy, right? Automatic garbage collection is important, right? They wouldn't turn it on by default if it wasn't!? Eh, not really. Not for you. I'm not being sarcastic. You just probably don't work at Meta or Microsoft scale.

Garbage collection for git in most every use case just isn't worth it. That is unless you're very regularly committing very large files, and then orphaning those commits. I don't see that happening in most uses cases; garbage just doesn't build up that quickly in normal usage.

Don't believe me?

I have been working with the same git repo every day for ten solid years with gc turned off. My .git folder is currently 554 MB whereas a freshly cloned copy is 112 MB. I could be saving a whole 442 MB - whoop de doo.

I don't know about you, but I'll happily pay 442 MB for 10 years of improved data safety.

But why turn it off?

Having it on means detached commits could disappear at any time. It's a roll of the dice.

Turning off automatic GC means that anything you once committed will remain restorable, forever. If a rebase goes badly, you can always safely get back to the previous state of a now-orphaned commit. There is zero chance of it ever disappearing.

Basically, turning automatic GC off means any amount of history rewriting is always undoable.

Example

A simple call to git reflog will show you all recent commits, including ones that are no longer on any branch.

$ git reflog
98a9b6b HEAD@{1}: commit (amend): Increase footer font-size and line-height
edc192c HEAD@{2}: commit: Increase footer line height
0360c78 HEAD@{3}: commit: Update footer color to quiet lighthouse
9231f91 HEAD@{4}: commit: Burn down a bunch of old css
…

In the example above, commit edc192c became 98a9b6b when it was amended.

To undo that amend, I can reset the HEAD of my current branch back to edc192c. I have no fear of edc192c ever randomly disappearing before I get around to it.

Turning off GC is a safety net that has saved me many times.

Turning off automatic GC means you could safely work on a detached HEAD if you're wild like that. You probably shouldn't do that unless you know exactly what you're doing, but you certainly could.

And of course…

Keep in mind, you can always just run GC manually if you feel like a lot of garbage has piled up.

$ git gc

I have seen many very large projects including Google owned projects get it wrong.

I brought the issue up at my local Go meetup, and no one had ever heard about the rule. They were very skeptical the rule existed at all.

A little history

For a long time Go did not include a built in method for versioning dependencies. It did include the sometimes controversial go get method of fetching packages. The go get tool until recently simply cloned the HEAD of a packages primary branch based on URL and placed it in your $GOPATH.

In what has been oft linked to Go's "extremist position" on backwards compatibility, the expectation was largely that if you published a library, you were expected to maintain compatibility when at all possible.

A plethora of tools popped up over the years to fill this void, gopkg.in, glide, and even the at one time "official experiment" dep.

dep was the heir apparent to dependency management with backing from Google.

Seemingly suddenly however Russ "rsc" Cox sprung vgo (aka Go Modules) onto the world as the official Go solution. There was seemingly no input from the community in its creation, to the chagrin of many.

Whereas dep was a largely standard dependency manager in the vein of npm and the like, Go modules is a decidedly opinionated "Go" solution. It's handled as part of the language as a build step. When you go build if you are missing the versioned dependency, the build tool will fetch it.

The dependencies requirements are listed in go.mod and the exact versions expected are in go.sum. Both files are in their own sort of domain specific language. The prior has a syntax reminiscent of Go itself. The latter is a space separated list of dependency versions largely not intended for human consumption.

The build step will add to these files as necessary, and running go mod tidy will clean them up.

The v2+ problem in detail

Go modules place a very strange requirement on package developers. When a module hits major version 2 or higher, the module name must end in the major version. The advantage to this is it creates a separate package. The result is that a project can now depend on multiple major versions of the same library.

There are two tactics to achieving /v2 modules:

The first method is to change the module name in your go.mod

An example may seen in the go.mod of github.com/google/go-github seems simple enough, but requires you to also change any cross references within your package. While not the end of the world, it is error prone and even Google missed some.

The biggest problem with this method is that it breaks subpackages in non-module-aware versions of Go. As time goes by this becomes less important, but in older versions of Go the package name had to exactly match it's location. https://github.com/pkg/foo could not be github.com/pkg/foo/v2 when there is no v2 directory. The Go Team acknowledging this trouble and patched rudimentary module support into point releases 1.9.7 and 1.10.3 of the Go language to help ease the transition.

The second method and the method the Go Team themselves promote is leaving your v1 project as is in the root of your repo, and actually creating a v2/ subfolder containing your v2 library.

This feels weird, but the advantage is it works cleanly with older versions of Go that do not understand modules. Off the top of my head I only knew of a single project that went this route, and they appear to have gone back on it.

For what it's worth, the official Go Blog tried to clear the situation up some, and posted a write-up about it.

blog.golang.org/v2-go-modules

Even as a seasoned developer, I found this write-up somewhat impenetrable. For how important and unusual of a requirement this is, the write-up is not accessible enough.

Many projects including Buildkite's terminal-to-html simply didn't know about the requirement until they forget to handle it, and end up publishing broken releases. A v2+ Go library with a go.mod file, but not respecting the module v2+ rules is a compile time error for dependants. This is worse for users than not being a Go module at all.

Some projects like GORM sidestep the issue entirely by tagging their 2.0 releases as far flung 1.x releases. That something of a solution, but smells terrible.

What's to be done?

First, I think the Go Team needs to do a better job of shining a light on this rule. They need to scream it from the rooftops. They need at the very least a section of the documentation that lays it out clearly and in layman's terms.

Beyond this, there's room for warnings from the Go tooling. Subpackages importing older versions of the root package should throw a warning. It's way too easy an issue to miss.

There's room for similar notes on godoc.org / pkg.go.dev, although the usefulness is questionable.

The best and least likely option would be for Go to drop the requirement and make it optional. I believe this is a workable solution. Packages who want to allow users to include multiple versions can follow the v2 guidelines, and others don't need to.

An option I would personally promote, but is likely never going to happen, is to require /v0 and /v1 as well, such that you'd hit the issue right away rather than years into a project. Teach the convention early.

Discussion at: news.ycombinator.com/item?id=24429045

Here's a tool to safely encode any given input to Go string. It's written in Go using Go's own string encoder and compiled with WASM.

The code as well as a CLI version of it are available on GitHub.

It's not ideal, but until the whole SharedArrayBuffer debacle settles down, it's probably the best I can do right now.

Update October 17 2022: The mp4's should now always be compatible with QuickTime. While they were always compatible with Zoom, occasionally not being compatible with QuickTime was confusing some people.

Stuck in boring Zoom meetings? Have the perfect gif you want to use as your "Zoom Virtual Background", but Zoom won't let you use animated gifs?

Here's a tool to convert GIFs to MP4's that Zoom will accept!

Launch Tool

Previously, this page was just instructions for using ffmpeg yourself. The tool uses ffmpeg for you via WASM to make it easy.

There's an easy solution - using ffmpeg to convert your GIF to an MP4.

You can install ffmpeg with brew or other package manager if you're on a UNIX-like system. Otherwise downloads are available here.

Once you have ffmpeg installed just run the following in your terminal, replacing <gif-file.gif> with the path to your gif.

ffmpeg -i <gif-file.gif> -vf scale=w=640:h=360:force_original_aspect_ratio=increase output.mp4

The flags on the command specify a minimum size of 640 by 360, Zoom's minimum video size, while maintaining aspect ratio by increasing either dimension.

In 2013 I wrote an unintentionally inflammatory post: Go Binary Sizes Are Growing out of Control. I had imagined no one would read it, but it took off. I wrote it because I noticed my binaries getting larger as I upgraded versions of Go and found it curious.

Much has changed since then; biggest of all there have a handful of major releases of Go in that time.

I wanted to find out how much this had changed since my post went up. I used Docker to automate the process of compiling against major releases of Go. The exact code I used is on GitHub.

I hit a couple of hurdles. The oldest official version of Go on Docker Hub is 1.2 - which happens to be the version my previous post left off. Also, many of the programs from the previous comparison no longer compile in such old versions of Go.

Thus, consider this a standalone comparison rather than a direct continuation of the previous post.

I ended up going through most of my Go projects to find what would actually compile in ancient versions of Go.

The tools I compiled for the comparison are in order

• hello world using fmt
• hello world using println
• Hookah - a tool for GitHub automation
• imgavg - a tool for averaging a collection of images
• sqlread - a tool for querying MySQL dumps without loading them into a database.

The Results

Without further ado, the raw data in megabytes.

Summary

The trendlines show the majority of the projects have a small downtick in size. The Go 1.5 and Go 1.6 releases caused a pretty universal uptick which 1.7 seems to have largely mitigated.

Hookah shows by far the most fluctuation in size by a pretty large margin. This is interesting because comparing dependencies the only builtin packages Hookah uses that sqlread does not are regexp and regexp/syntax. sqlread itself actually utilizes a pretty large number more. This could indicates the variation is deeper than the changes to the built in libraries, and potentially more to do with the compilation output and optimization.

Everything except for the simplest "Hello World" hello-nofmt has shown an overall decrease. hello-nofmt boils down to println("Hello, 世界") has shown an overall 658kb increase. This is likely the purest illustratration the growth in size of the runtime.

Conclusion

Go binaries are not "growing out of control". They're pretty stable and for the most part have shown a decrease.

I'm still hopeful future optimizations will help improve sizes. Given WASM becoming a major compilation target, size matters more than ever as it directly affect load times.

Recently a new tool showed up called mkcert has come and made life way easier.

It's a wonderful tool that instead of building Self-Signed Certificates, sets your local machine up with a certificate signing authority and creates valid certificates.

You can read more about how it works at the link above. Needless to say, the process is much less painful than creating self signed certificates.

Installing mkcert

The easiest way to install mkcert is via Homebrew.

Beyond mkcert we should also install nss which helps mkcert set up Firefox to respect its certificates.

$ brew install mkcert nss

Once that's complete, we need mkcert to set itself up as a signing authority on our local machine.

$ mkcert -install

Creating the Certificates

Once we have mkcert set up, the next step is to generate the certificates for the intended local domain(s).

You may include multiple domains or wildcard subdomains. You can find more information about mkcert's capabilities here.

$ mkdir /tmp/crt && cd /tmp/crt
$ mkcert local.example.com

You should get an output something like as follows

Using the local CA at "/Users/user/Library/Application Support/mkcert" ✨

Created a new certificate valid for the following names ????
 - "local.example.com"

The certificate is at "./local.example.com.pem" and the key at "./local.example.com-key.pem" ✅

Now that we have our certificate, we can move it into the apache /etc directory:

$ sudo mkdir /etc/apache2/ssl
$ sudo mv *.pem /etc/apache2/ssl

Apache Configuration

Next, all we need to do is configure Apache.

Follow the Apache Configuration section's instructions, stopping and returning here before the "Chrome / Safari Configuration" section on my previous post.

When you reach the portion about configuring your VirtualHosts, the only difference is the filenames will look more akin to this pattern:

  SSLEngine on
  SSLCertificateFile "/private/etc/apache2/ssl/local.example.com.pem"
  SSLCertificateKeyFile "/private/etc/apache2/ssl/local.example.com-key.pem"

Unlike the prior walkthrough which restarted Apache with a reboot, we instead now only need to restart Apache.

$ sudo apachectl restart

And we're done! Unlike setting up a self signed certificate, we do not need to add anything to our keychain or browsers. We are ready to go, your certificate should work as expected.

While trying to compile PHP extensions after installing macOS Mojave, I was butting up against a ton of missing header file errors including php.h and zlib.

After several hours of beating my head against my desk and Googling/Stack Overflowing I found the solution.

Ends up you just need to force re-install the header files. Special thanks to @sfdye for the solution!

$ sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /

If you get an error about that file not existing, you probably simply don't have the Xcode command line tools installed which can be remedied simply as follows:

$ xcode-select --install

I recently purchased a .app domain for a side project; .app domains are interesting as they require SSL at all times.

I like to set up my dev environments as subdomains of my actual domain, such as local.donatstudios.com. It makes working with CORS and cookies a lot simpler for applications with APIs on subdomains.

Doing this with .app however means having to get SSL working. I found a number of tutorials of varying quality online, but none of them did the full process from start to finish, particularly solving problems with Chrome 58.

This tutorial assumes you are using the built in version of Apache that comes with macOS, however you should be able to adjust paths for brew or otherwise installed versions.

Creating the Certificates

First off we'll make place to do our work. We need to do this outside of sudo requiring areas due to some bugs with the Apple version of openssl.

$ mkdir /tmp/crt && cd /tmp/crt

Next we will generate your key and certificate. Replace all four local.example.com entries with your full intended development subdomain.

The subjectAltName=DNS piece of this is required to get Chrome versions greater than 58 working.

$ openssl req -newkey rsa:2048 -x509 -nodes \
    -keyout local.example.com.key \
    -new \
    -out local.example.com.crt \
    -subj /CN=local.example.com \
    -reqexts SAN \
    -extensions SAN \
    -config <(cat /System/Library/OpenSSL/openssl.cnf \
        <(printf '[SAN]\nsubjectAltName=DNS:local.example.com')) \
    -sha256 \
    -days 3650

Apache requires a nopass key, so to generate this we do the following, again adjusting paths as nessessary:

$ sudo openssl rsa -in local.example.com.key -out local.example.com.nopass.key

Now that we have completed the key generation, we can move our certificates into the apache /etc directory:

$ sudo mkdir /etc/apache2/ssl
$ sudo mv *.key *.crt /etc/apache2/ssl

Apache Configuration

In your favorite editor open /etc/apache2/httpd.conf

Uncomment the following lines (line numbers may vary):

Near line 89:

LoadModule socache_shmcb_module libexec/apache2/mod_socache_shmcb.so

Near line 144:

LoadModule ssl_module libexec/apache2/mod_ssl.so

Near line 528:

Include /private/etc/apache2/extra/httpd-ssl.conf

Next open /etc/apache2/extra/httpd-ssl.conf, find <VirtualHost _default_:443> around line 121.

Either comment out using a # at the start of each line, or delete every single line between the opening <VirtualHost _default_:443> and the closing </VirtualHost> around line 290.

Presuming your sites VirtualHost is in /etc/apache2/extra/httpd-vhosts.conf either change <VirtualHost *:80> to <VirtualHost *:443>, or duplicate your VirtualHost entry using <VirtualHost *:443> if you'd like your VirtualHost accessible with and without SSL.

Next, inside your sites <VirtualHost *:443> block, append the following, adjusting the filenames accordingly.

  SSLEngine on
  SSLCertificateFile "/private/etc/apache2/ssl/local.example.com.crt"
  SSLCertificateKeyFile "/private/etc/apache2/ssl/local.example.com.nopass.key"

Chrome / Safari Configuration

Next we will open your Apache ssl directory and the "Keychain Access" application:

$ open /etc/apache2/ssl
$ open /Applications/Utilities/Keychain\ Access.app

In the Keychain Access application, select the login Keychain and the Certificates Category.

Then from the open Finder window, drag and drop your .crt file.

Double click, and change the settings for the selected certificate to "Always Trust".

Lastly, repeat the process of dragging and giving access in the System Keychain.

You may need to reboot, but your certificate should now work in Chrome and Safari on your local machine.

To prevent developers from pulling down potentially unsafe packages, we now retire the namespace of any open source project that had more than 100 clones in the week leading up to the owner’s account being renamed or deleted.

This is a decent half step but I'd still love to see either permalinks or every namespace of a user permanently retired once their account is deleted.

Update 2018-02-10 In discussing this with people online, I’ve come to the conclusion that the bigger, more important issue is lack of permalinks to repository instances. Path reuse, rather than username reuse.

There is a very popular tool for embedding data files into your Go binaries called go-bindata. Several days ago however the user who ran it, "Jim Teeuwen" (wayback machine), completely disappeared from the internet – deleting his GitHub account and personal domain in the process.

While this is within his rights to do, this broke a dependency many people had within their projects.

To fix this, some users of the project recreated the account and the repository based on a fork of the project.

They have an official announcement/disclosure here:

https://github.com/jteeuwen/go-bindata/issues/5

At the very least they are being honest about it.

The fact that they were allowed to do this however represents a fundamental flaw in GitHub's security model.

Usernames, once deleted, should never be allowed to be valid again. Many sites including Google do it this way.

Allowing username reuse completely breaks any trust that what I pull is what it claims to be.

What if this user had been malicious? It may have taken a while before someone actually noticed this wasn't the original user and the code was doing something more than it claimed to.

While Go's go get functionality is no doubt naive and just pulls the head of a repository, this is not exclusively Go's problem as this affects any package manager that runs on tags. Simply tag malicious changes beyond the current release and it would be deployed to many users likely with little actual review.

This should not be possible. This is scary and should be fixed.

Update

Many people are arguing that this is the developers fault or the package managers fault. I do not agree, but as far as I see it that doesn't matter anyway. The simple fact of the matter is that it is being used like that, like it or not, and the simplest and I argue most correct fix is for GitHub to prevent the issue.

I think another good option would be GitHub offering permalinks to repos, such that if they were deleted and recreated the pathing would change.

It affects not only package managers and programs and software, but humans. Humans navigating GitHub. I have no way to tell while navigating the site if a project is the original or a charade. That is a problem.

As for the title change, I agreed with Hacker News that the original was a bit hyperbolic. I have a tendency to resort to hyperbole to get my emotional point across - and it doesn't always read correctly on the internet.

Static analysis makes up for the missing type system and allows you to scale.

I recently got in an argument with another developer over static analysis. It's been an ongoing point of contention. He was angry that code that works fine but didn't document its undetectable return types was failing CI. He argued that if the code works, it should pass. He argued that if the code works, it is "correct".

I put forward the exact opposite position. Code that works but doesn't statically analyze is incorrect. Code that is correct has no potential of runtime error. 

My intentions are not only to sell you on the idea that you should be statically analyzing your code, but to convince you that it is fundamental to responsible reliable development.

Humans are fallible; code written by humans doubly so. By it's very nature that code is broken until proven otherwise. If it cannot be statically analyzed, it cannot be proven correct.

If code runs as expected, this only proves it correct in the limited scope it's been ran it within. Be that ones, tens or even hundreds of thousands of parameter combinations, it's always limited. There therefore must be zero expectation of it continuing to work.

To the contrary, if code statically analyzes unerring, it will run without runtime error in all cases, save a failure of the analyzer. This code is proven. This code is correct.

Code which "works" does nothing to prove that it's used as you expect. In particular elsewhere within the project. It does nothing to prove it will remain correctly used in all cases in the future.

I know what you are thinking "But… but… my code runs correctly within my expected domain", and that's fine. That is what's expected of it. But throw exceptions if it happen it doesn't know how to handle, things that creep outside the expected domain. It lets the future developers, the analyzer, and the program itself know the intended domain and when it steps outside it. 

No amount of fuzz testing can account for another developer doing something you never expected.

To give a trivial but near real life example, consider the following:

function getUserById( userId ) {
    if(userId > 0) {
        return new User($userId);
    }

    return null;
}

var user = getUserById( input.value );
console.log( "Hello userId: " + user.getId() );

The above code may work throughout all your testing. Every place you use it, every time you try it, everything your application throws at it - success. 

This code works, and yet this code is broken. This code does not handle the potential null, and has the potential to trigger a runtime error. This code has the capability of a runtime error, and is therefore incorrect.

Making this code correct would be as simple as:

var user = getUserById( input.value );

if(!user instanceof User) {
    throw new Error("User not found");
}

console.log( "Hello userId: " + user.getId() );

An Exception, as opposed to a runtime error, is something the code and coder anticipates. It's an easy way to show you've thought about the domain of your code. Even unhandled, it's clear the developer considered the possibility. It's easily recoverable. Throwing an exception here is correct.

Obviously you'd probably want to handle that Error in the example somewhere, but that's outside the scope of this writing.

The trivial case above is reasonably obvious and easy to spot. Let's instead imagine the function to get a user is deep in your codebase. While we are at it, let's also say that the logical User|null union we've created gets passed around. Passed several levels deep, rather than using it immediately where it's fetched. It rapidly becomes much less obvious, nigh impossible, to notice that there is an issue. A call 8 methods deep has no idea it might be receiving a null. This is exactly the kind of thing a static analysis is for.

A static analyzer knows the full scope and range of possible values. It is able to pinpoint problems a human fails to notice. Noting all function calls within the scope of a project and knowing all possible.

All this withstanding, static analysis cannot prove the logic is correct. That's not what static analysis is for. Unit tests and other testing prove that. Static analysis proves that the code itself executes free of unhandled runtime errors. It is the type checker your language forgot.

Arguable the better solution is to switch to a type safe language; one with compile time type checks and ideally union types. That's not always an option for most people though, consider the recent meteoric growth of JavaScript. 

TypeScript is without a doubt the best option for JavaScript. I highly recommend it — consider it static analysis on steroids. It's JavaScript with the addition of strong typing. There's no reason to be writing straight JavaScript anymore. TypeScript is really a wonderful all in one solution.

Elm is a neat alternative but a completely different paradigm. While it compiles to JavaScript, it's completely foreign and more of an all-in situation. It borrows much of its syntax from Haskell, and offers the aforementioned union types. 

For PHP on the other hand I recommend a handful of tools. Scrutinizer is in my experience the most feature complete of the hosted CI services. I make extensive use of PHP CodeSniffer which is an amazing tool based around a PHP tokenizer. It's a lot of fun writing your own sniffs in my experience. I have begun experimenting with Phan - it's a little unforgiving, in a good way… And of course PHPStorm has a pretty decent analyzer built in.

Partially as a companion piece to my recent post about how CSV is an encoding nightmare, and partially an expression of frustration, I've decided to make a list of falsehoods programmers believe about CSVs. I recommend my previous post for a more in-depth coverage on the pains of CSVs encodings and how the default tooling (Excel) will ruin your day.

Everything on this list is a false assumption that developers make.

1. All CSVs are ASCII
2. All CSVs are Win1252
3. All CSVs are in 8-bit encodings
4. All CSVs are UTF-8
5. All CSVs are UTF-16
6. All CSVs contains a single consistent encoding
7. All records contain a single consistent encoding
8. All fields contain a single consistent encoding
9. All CSVs contain records
10. All records contain fields
11. Fields never contain record separators
12. Fields never contain delimiters
13. Fields never contain control characters
14. Delimiters are escaped with a \
15. All fields are enclosed by double quotes
16. All records are a single line
17. All lines contain a single record
18. All records contain the same number of fields
19. All records contain the same number of fields as the header
20. All records contain the same number of fields or fewer than the header
21. All CSVs contain a header
22. All record separators are CRLF
23. All record separators are LF
24. All record separators are a single byte
25. All record separators are a single rune
26. All newlines are a single byte
27. All CSVs are delimited with a comma
28. All CSVs are delimited with a comma, tab or semicolon
29. TSV isn't CSV
30. All delimiters are a single byte
31. All commas are a single byte
32. All CSVs are readable with Excel
33. Excel is a good tool for working with CSVs
34. Excel is an OK tool for working with CSVs
35. Excel can losslessly save CSVs it opens
36. Using ="{value}" is a good way to get around Excel auto-formatting
37. The first line will never be a poorly supported instruction header
38. Using sep={char} is a good way to get Excel to accept your delimiter
39. Prepending a BOM is a good way to get Excel to read your encoding
40. You can safely name your first column "ID"
41. All CSVs follow RFC4180
42. Most CSVs follow RFC4180
43. All CSVs follow the same defined standard
44. All CSVs follow a defined standard
45. All CSVs have a .csv extension
46. All CSV is human readable

Please take these into consideration next time you find yourself working with CSV. If you can think of anything I may have missed I'd be happy to add it.

As a suggested further reading, "The Art of Unix Programming" http://www.catb.org/esr/writings/taoup/html/ch05s02.html#id2901882 section on DSV style which notably says "CSV is a textbook example of how not to design a textual file format"

Updates:

• Thanks to Max for pointing out the Excel supported sep={value} header I was strangely entirely unaware of.
• Thanks to Don Hopkins for the note about not being able to start a header with ID

It would appear in a recent update Microsoft has added support for safely reading and writing UTF-8 CSVs to Excel. There is a new format in the save dialog CSV UTF-8 (Comma delimited) which is distinct from Comma Separated Values which is also still in there.

In my testing it appears to safely both load and save. Best of all it appears if there is a BOM, it leaves it, if there isn't it doesn't add one. Very nice handling indeed.

UPDATE 2022-07-22:

It would appear that the stricken above was incorrect. Current versions of Excel will only reliably load a UTF-8 CSV if a UTF-8 BOM is in place.

The company I work for manages a self-service data import system, handling information from school districts “Student Information Systems”. There are hundreds if not thousands of SIS’s out there, but what the vast majority have in common is the ability to export CSV. It is the lingua franca, everyone can get us CSVs.

CSV to the casual observer seems a simple portable format, but its looks are deceiving. If the data is pure ASCII (bytes 0-127) you’ll be fine. However, if there is a need for any sort of non-ASCII character, there is some work ahead.

In our case, handling school data from around the world, correctly handling non-ASCII characters is of the utmost importance.

Excel: A World of Hurt

The biggest problem is not CSV itself, but that the primary tool used to interact with it is Excel. Excel handles CSV encodings badly.

Creating a new document in Excel and saving as ”Comma Separated Values (.csv)” it uses your locale’s Windows or Mac codepage. Win-1252 and MacRoman respectively in the United States. If your codepage doesn’t support a character in your document, it will be silently replaced with an underscore _ character.

Because it uses codepages and not a Unicode encoding, it makes processing a painful chore. There is no way to tell the difference between different 8-bit codepages programatically.

One can use heuristics to sort them into an order of likelihood, but there is no way to ever know for sure. We chose to present the user with previews of the most likely codepages, and let them pick the correct one.

Excel for Mac: Broken Beyond Belief

The Macintosh version of Microsoft Excel is particularly harrowing.

As I mentioned above, Excel saves your CSV in your locales codepage. One should note that the Mac codepages fell out of use with OS 9, almost 15 years ago. Microsoft did not get that memo.

While that by definition makes the CSVs the Mac version of Excel exports unusable on Windows, the problem is more unfortunate than that. The Mac version can only read the locales Windows codepage.

That means the Mac version of Excel cannot read CSVs it wrote. That's pathetic.

If you had any extended characters when you saved, they are scrambled when you reopen it. This problem has persisted in every version of Mac Excel up to the current Excel 2016.

One simply cannot safely open a CSV created with the Mac version of Excel, on any platform, anywhere.

UTF-8? What’s that?

UTF-8 is the encoding of the 21st Century. It was quick in overtaking every other encoding. Almost every modern application supports it. This will be easy and we’ll be fine? Not so fast.

If one attempts to open a CSV file encoded as UTF-8 without a Byte Order Mark (BOM) as recommended, any non-ASCII characters are again scrambled.

This is the CSV format Apple’s Numbers exports by default, UTF-8 sans BOM.

If we try it again with a UTF-8 BOM prepended to the file and Excel will read it. This is deceptive because once saved the text will remain correctly encoded UTF-8, but bizarrely the BOM will be stripped causing the file to no longer be correctly readable.

Many naïve application output UTF-8 + BOM CSVs, and they read correctly but do not write correctly. I’ve seen this cause all sorts of headaches because it appears to work but does not.

The Right Way Functional Workaround

As of this writing, there exists a single usable CSV format that Microsoft Excel can both read and write safely across platforms. Tab delimited UTF-16LE with leading Byte Order Mark.

Using Excel to output into this format to begin with, you have to use the Save As dialog and choose “UTF-16 Unicode Text (.txt)”. Frustratingly, it includes a .txt extension by default which can be confusing for non-technical users. However, if you provide clear instructions to Windows users to save their filename in quotes with a CSV extension such as "example.csv" it is workable.

If you operate an application with a CSV exporter please use this as your default export format. It will save us all many headaches in the long run.

Reading Hacker News today there is an article about how OS X Installers stopped working on February 14th, 2016. This is due to an expired certificate Apple used to sign the binaries with.

In that same time range I had a handful of Apps purchased from the App store suddenly and mysteriously stop working. They just quit. No dialog, no error mesasage, nothing at all. Apps I paid for and depend on just stopped working.

The thing they all these Apps have in common is they are no longer available from the App store and have not been updated in several years. I am nearly certain that this is the same signing issue.

If this is the case, this means all Apps from the app store are signed built-in self destruct date. One which only Apple or the original author can fix.

I have been careful to keep backups of the .app’s. Apple dropped the ability to re-download App's removed from the App store.

This gives way too much power over my work to third parties. Software I paid for should not stop working because someone else stopped supporting it. How can someone build a reliable work-flow when any program stop working at any time? You can't, that’s simply insanity.

One of the apps I used daily, the author took down a few years ago because it “flopped and [was] plagued with technical issues”. I have encountered no technical issues with it until this point, and it an important part of my work-flow.

I, as well as others have asked the author to open source it on many occasions. [1], [2], [3], [4] … There’s more

Her only response to our cries has been a cringe-worthy LOL. I wish I were joking.

The moral of my story comes down to the following life lessons.

• Don’t depend on things that others can take from you.
• Be able to scratch your own itch; seek open source when at all possible.

I will no longer be purchasing things from the Mac App Store. The convenience is not worth the incredible down sides.

Richard Stallman has been warning us for years, I should have listened.

Searching my library, I found 74 tracks that had this issue.

I found this irritating and inaccurate, so I set forth finding a way to correct it.

I wrote the following simple AppleScript to set the playcount of such tracks to 1.

All you need to do is either paste the following into the 'Script Editor' or download it and run it. See video below if you need help running it once it’s open.

Underscore.js released a second digit "minor update" that was actually a breaking update for some people. They argue it was minor because not very many people would be affected. The community however was angry.

On the opposite side of things, PHP Internals announced they will be skipping version 6 and numbering the next version of PHP 7 to avoid confusion with the never released UTF-16 native version. Again the community was angry.

Using Version Numbers Well

If you are semantically versioning your library, the version represents a contract of trust between you and your developers regarding the public API. The contract declares that nothing you change in a 'less than major' release should break their code. Ever.

Break the contract, you can end up breaking their trust. Break their trust and they'll look elsewhere.

Version Numbers SHOULD NEVER Be Branding

If you're using your version numbers as branding, you are doing it wrong. Version numbers should show a clear history of the public facing interface, not what you sell or advertise it as.

Microsoft of all people understands this. While their version numbers aren't truly semantic, they are aimed at developers - and highly out of sync with the marketing designation because the version numbers denote a true relationship.

You can have both a release number and a sales version. This is the way to go if sales demands control of the version number, such as where I work.

Increment Early, Increment Often

Breaking changes always need to be a first digit release. Always. It doesn't matter if it's not a common use case. It shouldn't matter if you're saving the number for a "woo big new release" version. A breaking change in a minor release violates trust and brings the quality of your software into question.

WHENEVER a behavior or signature of ANY explicitly public member of your API has changed whereas the previous behavior was anything other than bold face wrong it is a first digit release.

Numbers are Cheap

Numbers are cheap. Numbers are infinite. Never reuse numbers, ever. There is no reason.

PHP in my strong opinion did the right thing by skipping 6. They sidestepped ANY potential confusion, minor as it may have been. The argument that they caused confusion about "Where did 6 go" is childish. People will be investing time in the largest digit release, regardless of gaps. No one (save some overly cautious IT people) is going to be using PHP 5.6 and on seeing 7 released think "I'll upgrade to 6".

Trying to not have gaps in your version numbers makes as much sense as if Git were to use ordered numbers instead of hashes.

Version numbers should be a universally unique identifier for a specific set of code. If there is ever any form of contention over what a version number represents, increment. Don't think about it - just do it.

Conclusion

Always avoid confusion; don't ever be afraid to increment your major release number. Only good things can come from it.

Trust is maintained.

It would seem that very similar but non-identical colors seem to pop up often in CSS files of any reasonable age, and I became sick of them. This started as a little script to help me find them in a stylesheet, and grew into this full-fledged tool.

This tool will scan CSS and CSS-like files like Sass, Less, etc and find colors similar to each other within a set tolerance.

This handles #hex, rgb, rgba, hsl, and hsla colors.

There is a command line version of this with Continuous Integration support available on GitHub.

These directions are for working with the native installation of PHP. Your results may vary if you are using a brew, MAMP or otherwise installed version of PHP - I do not recommend this for those cases.

Important: As you are altering the built-in version of PHP, you will need to ensure you have disabled System Integrity Protection before you begin. You can find instructions on how to do this here: How to: Disable macOS System Integrity Protection.

This was previously dedicated to installing the Mcrypt extension specifically, but in reality can be used to install any of the following extensions distributed with the PHP source.

bcmath       ftp          mcrypt       pdo_oci      simplexml    wddx
bz2          gd           mysqli       pdo_odbc     skeleton     xml
calendar     gettext      mysqlnd      pdo_pgsql    snmp         xmlreader
com_dotnet   gmp          oci8         pdo_sqlite   soap         xmlrpc
ctype        hash         odbc         pgsql        sockets      xmlwriter
curl         iconv        opcache      phar         spl          xsl
date         imap         openssl      posix        sqlite3      zip
dba          interbase    pcntl        pspell       standard     zlib
dom          intl         pcre         readline     sysvmsg
enchant      json         pdo          recode       sysvsem
exif         ldap         pdo_dblib    reflection   sysvshm
fileinfo     libxml       pdo_firebird session      tidy
filter       mbstring     pdo_mysql    shmop        tokenizer

We need to install the required dependencies. If you are not already using Homebrew you will need it.

$ brew install autoconf pkg-config

For certain extensions like mcrypt you may additionally need to install additional libraries such as:

$ brew install libmcrypt

Next we will download the PHP source. Verify the exact version of PHP you are running. This can be retrieved as follows. The version is highlighted.

$ php --version
PHP 7.1.19 (cli) (built: Aug 17 2018 18:03:17) ( NTS )
Copyright (c) 1997-2018 The PHP Group

Now we move into a working directory and download the source making sure to update the following for the version from above.

$ cd /tmp
$ curl -L http://php.net/get/php-{{php-version}}.tar.bz2/from/this/mirror > php.tar.bz2
$ open php.tar.bz2

Now we will compile and test the extension.

$ cd php-{{php-version}}/ext/{{extension}}
$ phpize
$ ./configure
$ make
$ make test
$ sudo make install

If all that goes well, finally we'll need to add the following to our php.ini - I usually add at it at the end of the file.

extension = {{extension}}.so

You can verify your installation with the following:

$ php --info | grep {{extension}}\\.

Lastly, depending on your setup now, you may want to restart Apache.

$ sudo apachectl restart

The Setup

For the following examples, {username} represents your username.

Firstly, if you don't already have a Sites folder, create one in your home directory.

$ mkdir ~/Sites

Create or update /etc/apache2/users/{username}.conf to look as follows:

<Directory "/Users/{username}/Sites/">
    Options Indexes MultiViews FollowSymLinks
    Require all granted
    AllowOverride All
    Order allow,deny
    Allow from all
</Directory>

Add your virtual hosts if you don't already have them to /etc/apache2/extra/httpd-vhosts.conf

ala:

<VirtualHost *:80>
  ServerName localhost
  DocumentRoot /Users/{username}/Sites/
</VirtualHost>

Configuration

/etc/apache2/httpd.conf

Search for and uncomment the following lines:

LoadModule deflate_module libexec/apache2/mod_deflate.so
LoadModule userdir_module libexec/apache2/mod_userdir.so
LoadModule rewrite_module libexec/apache2/mod_rewrite.so
LoadModule php5_module libexec/apache2/libphp5.so

Include /private/etc/apache2/extra/httpd-userdir.conf
Include /private/etc/apache2/extra/httpd-vhosts.conf

Then around line 271 add index.php to the DirectoryIndex

<IfModule dir_module>
    DirectoryIndex index.html index.php
</IfModule>

/etc/apache2/extra/httpd-userdir.conf

Uncomment the following line:

Include /private/etc/apache2/users/*.conf

Almost Done

Restart apache

$ sudo apachectl restart

open a browser and test it out.

I for instance wanted to use a library only distributed by Zip.

It's actually fairly easy! In your composer.json file, you simply add a repositories section to the root. You will want to update the url section to point at the remote zip you intend to load, as well as updating the name and autoload sections. More information on configuring composers autoloader can be found here.

"repositories": [
    {
        "type": "package",
        "package": {
            "name": "dr-que/x-y",
            "version": "master",
            "dist": {
                "type": "zip",
                "url": "http://xyplot.drque.net/Downloads/XY_Plot-1.4.zip",
                "reference": "master"
            },
            "autoload": {
                "classmap": ["."]
            }
        }
    }
]

Then in your require section, add your selected name as follows.

"require": {
    "dr-que/x-y": "dev-master"
}

Then just run composer update and you should be ready to roll!

While doing code reviews on GitHub I find myself profiling a lot of SQL queries, getting EXPLAIN output into a markdown format for GitHub became very important, and was a huge pain to do by hand.

While there were tools out there, nothing I could easily copy and paste tab seperated query results into, so I created this. The markdown this outputs should work with most table supporting Markdowns flavors such as Markdown Extra and GitHub Flavored Markdown.

The default tab separated setting is very useful for pasting straight from Excel or other tabular data sources like SQL editors.

Caveat - while it does essentially parse CSV it is not quotation mark aware. If need or demand arises I may look into it.

The script is available on GitHub and is Node.js ready. More details can be found on GitHub. It is also available via npm.

Disable HTML 5 Validation

Clicking this will disable html5 validation for all form elements on the page. Useful for testing server side validation while leaving HTML 5 validation in place.

Element Count

Clicking this will provide a count of the number of elements on the page in an alert box.

If you're not taking advantage of __invoke, you should be. Why? It provides a uniform execution points for objects that have a doPrimaryAction() style method.

What do I mean by this? Many simple, single responsibility objects have a usage that goes something like:

1. Instantiate
2. Set Options
3. Execute
4. [Optional] GOTO 2

They are essentially glorified functions with manageable defaults.

So instead of

<?php

public function add($left = 0, $right = 0) {
    return $left + $right;
}

echo add(1, 2); // 3
echo add(3, 2); // 5

We have something like

<?php

class Add {
    public $left;
    public $right;

    public function construct($left = 0, $right = 0) {
        $this->left  = $left;
        $this->right = $right;
    }

    public function doMath() {
        return $this->left + $this->right;
    }
}

$adder = new Add;

$adder->left  = 1;
$adder->right = 2;
echo $adder->doMath(); // 3

$adder->left  = 3;
echo $adder->doMath(); // 5

The example is farcical and stands to illustrate my point. Everything except the doMath() method is setup for the doMath method. You are setting up the environment for doMath to execute.

This pattern of object with one primary action has a downside over a function. You need to know the class name and the invoking method name, which varies from object to object. This dear reader is where __invoke comes in.

__invoke is a magic method allowing us to treat our object as a function or more correctly a closure. It will pass the Callable type hint when added, and makes the object directly invokable via call_user_function.

What this gives us most importantly though is a uniform way across differently intentioned objects to invoke its primary action.

If we update the previous example:

<?php

class Add {
    public $left;
    public $right;

    public function construct($left = 0, $right = 0) {
        $this->left  = $left;
        $this->right = $right;
    }

    public function __invoke() {
        return $this->left + $this->right;
    }
}

$adder = new Add;

$adder->left  = 1;
$adder->right = 2;
echo $adder(); // 3

$adder->left  = 3;
echo $adder(); // 5

You can see the difference in use is slight, but the benefit is the lack of ->doMath. We know this object does one thing, now we have a uniform way to make our objects do that one thing.

I haven't seen a lot of traction with this in the community yet but I remain quite hopeful for this to pick up. I do know however that Aura appears to be a fan of the pattern, with Aura.Dispatcher using it beautifully.

• Aura.Dispatcher - Aura's Dispatcher has one primary purpose - to dispatch. As such, it is a perfect use case for being invokable.

One such example:

The PHP community doesn't care about things like lambdas, they just care about getting a site up as fast as possible.

This is just rubbish. PHP got the lambda treatment in 2009, while Java received it only months ago. The speaker is a Java developer who walked away from PHP over five years ago. He had not seen it grow up. He doesn't know the current state of things, he is misrepresenting PHP.

Later on, he asked what technology the audience used. I loudly proclaimed PHP, despite the speakers previous grumblings. An audience member in front of me turned around and scoffed, or at least so I thought. I felt at least a little like a martyr at this point and decided to take action. Justice was in order.

After the talk, the "bearded dude" approached me. He had seen the tweet thanks to the hash tag. He apologized and said it was not a scoff, but rather an expression of excitement that someone would bring it up after the speakers treatment of the language. He works in the language some too and was glad to see someone stand up for it. I shook his hand and apologized for the tweet.

Later on he left the following reply:

I had misjudged him, as the speaker had misjudged PHP. Alas the hypocrisy; I was no better than the speaker.

The whole experience got me thinking about why there is still so much FUD and ill will around PHP. After all, we've had many modern niceties for a good while now. The language has matured significantly in the last decade. It has become a modern language that's scalable, fast, and enjoyable, all while still easy to deploy.

Moreover, it has one of the best package managers I've ever used - Composer, and a vibrant and helpful community. Seriously, why all the hate?

The moral to this story is think before you act. Make sure you have the whole picture. I didn't, the speaker didn't. This has been my 2¢ for the day.

I've been testing the Alpha for many months now and thought I'd throw together some instructions for getting it going with git.

You can download the beta here: http://www.scootersoftware.com/beta.php

After you have the App installed, the next step is to install the command line tools. All other steps require this.

From the App, go to the "Beyond Compare" drop down menu and choose "Install Command Line Tools...". It will then prompt you for your password. After this, you bcomp file1.txt file2.txt

Using with Git CLI

The next step is to set it up as a git diff tool and merge tool. Git already has native support for it built in thanks to the Linux client having existed for years now.

This can be achieved simply by running the following commands.

$ git config --global diff.tool bc3
$ git config --global difftool.prompt false
$ git config --global merge.tool bc3

This simple setup will leave git diff and git merge functioning as usual, as well as allowing you to use Beyond Compare for the task by doing git difftool and git mergetool.

I highly recommend giving Beyond Compare a go as a merge tool next time you need to do a merge as it makes it wonderfully simple (when you understand whats going on with the 4 sections anyway).

Using with Tower 1.x

Update: Tower 2.0.6+ include built in support for Beyond Compare which the following instructions will interfere with.

1. Quit Tower
2. Navigate to ~/Library/Application\ Support/Tower
3. Create a CompareScripts folder if one does not already exist.
4. Download bcomp.sh and place it in the new CompareScripts folder.
5. Make it executable via chmod +x bcomp.sh
6. Download CompareTools.plist and place it in the ~/Library/Application\ Support/Tower directory. Rename / move any old version that might exist.

Using with SourceTree

1. Open SourceTree
2. Open "Preferences" from the SourceTree menu
3. Along the top bar, choose the "Diff" tab.
4. Under External Diff / Merge
   1. for Visual Diff Tool choose Other then in the Diff Command enter /usr/local/bin/bcomp and for Arguments enter $LOCAL $REMOTE
   2. For Merge Tool choose Other and in Merge Command enter /usr/local/bin/bcomp and for Arguments enter $LOCAL $REMOTE $BASE $MERGED

Note: This is not intended as a "Go Sucks" post. I love Go. I am not saying the developers are lazy or dumb or any of the things Reddit has implied. I am not implying I could build a better compiler. Rob Pike and the Go team are geniuses whom I look up to. The tone of this was intended to be a light "man I wish binaries were smaller" but that is not how Reddit seemed to take it. Also, the title is hyperbole and not meant to literally mean no one has control over it.

I started toying with Go about two years ago before the 1.0 release. I love the language, it meets me in a comfortable place between C and a scripting languages like PHP. I find it fun to write and an easy way to bust out performant code quickly. The binaries for a simple Hello, 世界 were large though, around 55KB. I figured this was something the Go team would work out later; it is a lot of space just to send a string to standard out.

With the release of Go 1.0 though, I noticed the binary size had grown slightly to 242KB. Discouraging but livable. I was quite hopeful with the 1.1 release, but it had actually nearly doubled in size coming out to a whopping 405KB. Then this morning I compiled it again in the newly release 1.2 and to my horror the ever simple program came out to 557KB.

What on earth is going on in that binary? I am no expert on compilers but I would imagine the vast majority of what is in there is completely unnecessary and could be optimized out. Changing the script to "Hello, World" rather than Go's flaunted unicode version has zero effect on binary size, as one would expect but I was hopeful.

For reference the complete source of my Hello, 世界 follows. Using fmt comes out even larger.

package main

func main() {
    println("Hello, 世界")
}

Here is a collection of binaries compiled in different versions of Go. Notice how imgavg grows from 1.9MB to 3.7MB. My applications are all small tools. I can only imagine the effect this has an already large application.

I really hope something is done about this. I would love to see tiny (<32KB) binaries from Go but right now that doesn't seem to be a priority for them. The fact that Hello, 世界 fills half a floppy means the language is not be useful for lighter or embedded environments.

As noted in the header there is an addendum to this:

• Go Binary Sizes Are Relatively Stable

But what if you wanted to load a single class from a git gist?

You can add the following repositories section to your composer.json file, adjusting name and url as necessary. The name should be a vendor/package (all lower) style name you make up for it, and url will be from the Clone this gist box on GitHub.

"repositories": [
    {
        "type":"package",
        "package": {
            "name": "turin86/wssoapclient",
            "version": "master",
            "source": {
                "url": "https://gist.github.com/5569152.git",
                "type": "git",
                "reference":"master"
            },
            "autoload": {
                "classmap": ["."]
            }
        }
    }
]

Then in your require section, add your selected name as follows.

"require": {
    "turin86/wssoapclient": "dev-master"
}

If the name of the class doesn't match the filename in a friendly autoload-able manner, you'll have to replace "classmap": ["."] with "files": ["ClassToAutoload.php"] replacing ClassToAutoload.php with the filename. More info on autoloading can be found here. Note that if you are experimenting with autoload settings you may need to remove your vendor directory and composer install, as updates to the autoload were not reflected for me after a composer update.

After this, just run composer update and voilà, you are loading a gist with Composer.

This tutorial only covers the default PHP installation that ships with Mac OS X / macOS. If you have installed a new installation this does not cover you.

For those using brew, you can simply brew install the appropriate brew formula for the extension.

These directions are verified to work with Mac OS X 10.9 Mavericks through macOS 10.14 Mojave.

$ csrutil disable

$ csrutil enable

The first step is to install the latest version of Xcode. On modern versions of MacOS this can be done from the App Store.

After this, install the Xcode developer tools.

$ xcode-select --install

Special note to Mojave users if the following steps fail, you may need to reinstall your Xcode header files. Instructions for that can be found here.

Next we will need to install the required dependencies before we can build the extension. If you are not already using Homebrew you should be.

$ brew install wget autoconf pkg-config libmemcached

You will also want to make sure you have PEAR installed; full instructions can be found here but can be summarized as follows:

$ wget http://pear.php.net/go-pear.phar
$ php go-pear.phar

Then in the /tmp directory we will create a folder we can work in.

$ cd /tmp
$ mkdir memcached-work
$ cd memcached-work

Next we will use pecl, part of pear, to fetch the current version of the extension.

The version of the memcached extension will vary so you will need to update the paths accordingly.

$ pecl download memcached
$ open memcached-{{version}}.tgz
$ cd memcached-{{version}}/memcached-{{version}}
$ phpize
$ ./configure
$ make
$ sudo make install

Finally you will need to add the following line to your php.ini

extension = memcached.so

You can verify your installation with the following:

$ php --info | grep memcached\\.

Depending on your setup now you may want to restart apache.

$ sudo apachectl restart

You should be all set to go!

To disable System Integrity Protection, boot into recovery mode by restarting and then holding ⌘R as you hear the startup chime. Then start the Terminal from the Utilities menu.

Run the following command

$ csrutil disable

Then reboot. You are good to go.

If you wish to turn it back on, follow the instructions above but instead use:

$ csrutil enable

After a coworkers departure, I inherited maintenance on our deployment bot, and it was written in CoffeeScript. While trying to make some small modifications, I ended up getting unexpected issues I didn't understand. Things I did not touch were suddenly breaking. I did not understand until I investigated the generated JavaScript.

It ends up CoffeeScripts approach to scoping of variables is to not scope variables. There is no shadowing. There is no var, let nor const equivalent, there is simply no way to explicitly define scope.

Variables in CoffeeScript are scoped to their outermost definition, regardless of whether the name was already in use elsewhere. When you use a variable of a name, its scope now includes all inner scopes, trampling any existing definitions of the variable in those scopes.

Let's examine the following examples:

test = (x) ->
  y = 10
  x + y;

alert test 5

var test;

test = function(x) {
  var y;
  y = 10;
  return x + y;
};

alert(test(5));

Now let's make one small change and define a variable y in the outer scope, outside the function.

y = 0;
test = (x) ->
  y = 10
  x + y;

alert test 5

var test, y;

y = 0;

test = function(x) {
  y = 10;
  return x + y;
};

alert(test(5));

The Problem

Notice the highlighting indicating the scope of the y variable. By defining y in the global scope all subsequent y's are also now that global. You may no longer have a variable named y in your code, at any deeper scope. The outer y is the only y.

Creating a global of a name given overrides all subsequent definitions.

In JavaScript var, let, and const keywords define the scope of a variable. Leave it out and it scopes downward until it hits a var definition of that variable or the global level. This means that inner scopes inherits from outer scopes unless it defines its own scope of the variable. CoffeeScript on the other hand offers you no method to scope a variable. Scoping is entirely left to the lowest level use of a variable name.

This behavior is dangerous and makes code very difficult to maintain. The problem is twofold. Firstly, when writing a function, you need to be aware of all shallower scopes variable definitions, as not to trample them. Secondly, when altering code at any scope but the outermost, you need to be aware of all deeper scopes variables as to not trample any of them.

The only "safe" way to write CoffeeScript is to assume all variable names are global. In practice they are.

There have been proposed solutions, such as a Go-lang style := for inner scoping, but these have not been accepted as of this writing. The creator of CoffeeScript, Jeremy Ashkenas, when questioned about it by Armin Ronacher on Twitter replied:

Not gonna happen ;) Forbidding shadowing altogether is a huge win, and a huge conceptual simplification.

This is not true for applications of any scale larger than toy. It makes development more complex, having to check every inner scope before using a new variable name. This is not simpler. The entire behaviour of all inner scopes can change with a single bad variable defintion.

Finally

CoffeeScripts goal of "expos[ing] the good parts of JavaScript in a simple way" was missed. Instead of improving JavaScript's scoping which can be kind of daunting to learn, they actually made it worse. If you want to improve JavaScript make every variable local until explicitly defined global, as is the behavior in the vast majority of languages.

JavaScripts "global unless defined as local" behavior is weird, but manageable. CoffeeScript's "local until defined elsewhere globally" behavior on the other hand is simply difficult to predict and makes it easy to accidentally change the entire meaning of code even when you entirely understand the behavior.

It is a bad behavior, period.

Further Reading

I'd like to also suggest reading Armin Ronacher's The Problem with Implicit Scoping in CoffeeScript. It eloquently describes the problem from a slightly different angle.

Update

• I've a fight on Reddit here.
• I've irked Reginald Braithwaite, the author of CoffeeScript Ristretto, so badly he's giving away 100 copies of his book for free!
  • The author has upgraded me to Not Wrong™!

A decade or so ago while helping a friends father clean out an old building, we came across an ancient Sun Microsystems server. We found it curious. Everything about it was different from what we were used to. The command line was black text on a white background, the connectors strange and foreign, and the keyboard layout was bizarre.

We never did much with it; turning it on made all the lights in his home dim, and our joint knowledge of UNIX was nonexistent. It sat in his bedroom for years supporting his television at the foot of his bed.

I never forgot that keyboard though. The thought that there was this alternative layout out there seemed intriguing to me.

I am ruined for all other keyboards…

Mac is the main platform at my new job. I found myself unhappy with Apple keyboards; they are flat, bland, and completely unsatisfying to type on. I dug out an old AppleDesign keyboard, and used that for several months but I still needed something more.

I read about Happy Hacking Keyboard, and after some deliberation decided to give it a try. I thought I could learn to accept most of the layout changes, but the lack of arrow keys on the "Professional" model made me squirm a little. Luckily there is a "Lite" version that includes arrow keys.

I purchased the Happy Hacking Keyboard Lite2.

The Happy Hacking Keyboard much like the Sun Microsystem keyboard I found years earlier features a UNIX layout. They designed it with typing at a terminal in mind.

A Better Layout

For those unfamiliar with UNIX keyboards, there are a few distinctions.

• The Control key moved to the Caps Lock keys usual position. This makes it much easier to hit common commands than its usual location, often without ever leaving home row. I can't imagine why anyone thought Caps Lock should have had this prominent of placement.

• The Meta (◆ / ⌘ / Windows) key is in its rightful place next to the space bar, as on a Mac keyboard.

• The Backspace key moved down a row to be above the Enter key. This change is my favorite change. Having a small reach, normally my hands have to leave home row to hit backspace. but on a UNIX board I have no issue hitting it with my pinky with the rest of my fingers remaining firmly on home row.

• Escape is also in reach of home row, placed where you would usually find ~. I could have done without this change, but it is fine.

• The ~ and \ have moved to the usual location of the Backspace. Working exclusively on UNIX systems now, I don't mind the \ being harder to reach. I could understand a Windows user finding this irritating.

The HHKB is a minimal version of the UNIX keyboard. To this end the F Keys are mapped to Fn + 1, 2, 3, etc. The HHKB encourages you to stay on home row; you can hit any key on the keyboard without leaving it. I find my typing accuracy and speed to have improved thanks to its use.

The HHKB features one more change I don't care for. The Backspace key by default performs Forward Delete. Luckily changing this is easy by flipping a toggle switch on the back of the board.

Closing

I recommend this keyboard to anyone who spends a lot of time at the terminal. You are able to do more of what you need without ever leaving home row, every important key is within reach.

Elsewhere

• Discussion on Hacker News
• Mentioned on BSD Now Podcast: Episode 333

I switched from Windows to Mac for work a little over a year ago (I've been using Macs at home for years) and there is one program I still could not live without - Scooter Software's wonderful Beyond Compare. I've tried every single diff tool available and none have lived up to Beyond Compare. It truly lives up to its name.

I've had Beyond Compare running in WineBottler for a while now, but was never able to get passing arguments to work correctly. Today I had a revelation. On creating the bottle it asks you for arguments to pass the executable, I had not provided any. Ends up it then passes the first argument as an empty string, which prevented the first argument being a file! Huzzah, an easy fix, pass it an innocuous argument! So here are some directions on getting it set up!

Beyond Compare Installation / Configuration

The first step is to download WineBottler if you don't already have it. Just drag the WineBottler.app and Wine.app into your /Applications directory. Then download the Windows version of the Beyond Compare Installer and hang onto it.

Launch WineBottler.app. In the left pane choose "Create Custom Prefixes", then for the Install File, choose the Beyond Compare Installer downloaded earlier. To my knowledge no Winetricks are necessary. Check the Self-contained checkbox to make a stand alone application.

IMPORTANT: WineBottler passes a blank argument if you leave the Runtime Arguments blank, which Beyond Compare interprets as the first filename - an annoying bug that took me months to figure out. To get around this we can pass it an innocuous argument (see arguments). I chose /nobackups because I don't want backups and they are turned off by default anyway.

Click Install. It will ask you to save your App, do so in /Applications. I named mine as BeyondCompare.app and paths later will reflect this.

Follow the installer. After the installer completes Beyond Compare may pop up if you've left the launch checkbox checked. Just close the window. Choose BCompare.exe when it asks you for the default executable (BComp.exe is for batch scripting).

Git Configuration

The paths in the following presume you named your app BeyondCompare.app so these may be adjusted as necessary.

Add the following to your ~/.gitconfig file. Remove anything existing that contradicts these settings. I've named the diff and merge tools bc3wb as git already has baked in support for the Windows version, which it calls bc3.

Beyond Compare 3 Professional

[diff]
    tool = bc3wb
[difftool "bc3wb"]
    cmd = /Applications/BeyondCompare.app/Contents/MacOS/WineBottlerStarter \"$LOCAL\" \"$PWD/$REMOTE\"
[merge]
    tool = bc3wb
[mergetool "bc3wb"]
    cmd = /Applications/BeyondCompare.app/Contents/MacOS/WineBottlerStarter \"$PWD/$LOCAL\" \"$PWD/$REMOTE\" \"$PWD/$BASE\" \"$PWD/$MERGED\"
    keepBackup = false
    trustExitCode = false

Beyond Compare 2 / 3 Standard

[diff]
    tool = bc3wb
[difftool "bc3wb"]
    cmd = /Applications/BeyondCompare.app/Contents/MacOS/WineBottlerStarter \"$LOCAL\" \"$PWD/$REMOTE\"

Note: I am unable to get the mergetool portion working in Beyond Compare 2 or 3 Standard as WineBottler is interpreting /savepath= as a filepath and converting it to z:\savepath= rather than properly passing the arguments. If anyone has any advice on how to fix this I would be ecstatic.

Voila, now you can git mergetool and git difftool it up!

The Problem

The following code example illustrates the issue.

<?php

class foo {
    function __call($name, $args){
        echo "__call " . $name . PHP_EOL;
    }

    public static function __callStatic($name, $args) {
        echo "__callStatic " . $name . PHP_EOL;
    }

    public function make_call_self() {
        self::ted();
    }

    public function make_call_static() {
        static::ted();
    }

    function make_call_classname() {
        foo::bob();
    }
}

$x = new foo();
$x->make_call_self();
$x->make_call_static();
$x->make_call_classname();
foo::sam();

The above returns the following when executed in PHP 5.3+ (excluding PHP 5.3.3, we'll get to this later)

__call ted
__call ted
__call bob
__callStatic sam

See Example

As you can see, all of those ideally should have triggered __callStatic but all save the last foo:::sam() triggered call. If you click the link to the example execution above, it shows that this did in fact work in PHP 5.3.3 but has since been broken.

I was just about to post this as a bug on PHP.net, but my friend Joel Clermont brought to my attention several cases of this being submitted in the past. One example being fairly infamous. Remember how I mentioned it working in 5.3.3? It was actually fixed in that release in response to a bug report. The problem apparently is that it broke a lot of things to do with the at the time new late static binding, and was rolled back in the next release.

From the horses mouth, via colder@php.net

You're misunderstanding the meaning of static::foo();

it doesn't mean "call foo statically" but rather "use runtime information to get the class currently called", and do a static(or not) call to the method of that class.

So static doesn't mean static apparently.

Understanding

After quite a bit of reading and a fair deal of explination by my friend Joel, I came to understand that ::, unlike C++, doesn't mean static at all. Rather it is a "Scope Resolution Operator" which determines scope from "context" meaning that when called from within an instance of an object, it makes instance calls if possible, rather than static calls. Infact, in 5.3+ you can even do $this::. The consequence of this is that self:: nor static:: are necessarily static calls.

This all rubs me wrong; I don't see the benefit to this behavior on the users end. Perhaps there is some backend optimization this allows. Even if that were the case I still feel like you could short circuit undefined self:: calls to __callStatic without breaking it.

Solutions

There were several "solutions" given, none of which are ideal.

• Call __callStatic directly.
• Use forward_static_call.
• Call the class from within a lambda. (After some testing I see that this does not work in PHP 5.4+)

None of these are particularly clean nor ideal. Instead I would like PHP to either add a new call like stat:: or simply fix self:: and static:: to trigger the proper call. The latter of course would be ideal, especially as static has the word static in its name.

See

• https://bugs.php.net/bug.php?id=62333
• https://bugs.php.net/bug.php?id=45159

To test what all behaves in this manner I threw together this simple set of three tests including instance members, static members, and class constants.

<?php

class foo {
    public $aaa = TEST_CONSTANT;
}

class bar {
    static $bbb = TEST_CONSTANT;
}

class baz {
    const ccc = TEST_CONSTANT;
}

define('TEST_CONSTANT', "Weird Result\n");

$x = new foo();

echo $x->aaa;
echo bar::$bbb;
echo baz::ccc;

Returns the following:

Weird Result
Weird Result
Weird Result

They all behave in the same manner, indicating that the members and class constants are not defined until the classes are invoked.

This is further demonstrated by the following example:

<?php

class foo {
    public $aaa = TEST_CONSTANT;
}

$x = new foo();
echo $x->aaa;

define('TEST_CONSTANT', "Weird Result. \n");

$y = new foo();
echo $y->aaa;

Which returns:

TEST_CONSTANTTEST_CONSTANT

So by touching the foo class, its initial value for $aaa has been locked in at what TEST_CONSTANT is at that time. Some testing indicates this also stands true for static methods and class constants as well.

This is a strange behavior I wanted to draw attention to. I certainly wouldn’t try to write something that was dependent on it, although our application currently seems to be.

For the purposes of this post I, me and all other pronouns referring to myself refer to a hiring professional.

Some Basics Up Front

Google yourself. Google your email address, I will! There will be lots of Googling before you're hired. I am well versed in the darker Google arts, and odds are I will find you. If I don't find you, that's almost as detrimental as finding something fairly negative; you are interviewing for a Web related job, but don't have a presence on the Web?

I will look at your Facebook / Twitter if they're open to the public. This can actually be a very good thing if your posts are nerdy.

Try not to have the top result for your name be the police report for the night you spent in jail. This is strangely common in my experience. Demonstrate some SEO skills, buy yourname.name! Do something to get that big red mark off the first page.

One of the most important things I can recommend is having a website. Not only will it ideally outrank your jail time on Google presuming your name's all over it, but ideally it's a technology website where you talk about experiences and things you've learned you want to share with others. This helps support your resume and your interview. This is very important, it brings substance to the game. It shows your interest in technology goes beyond making money. Also, lots of bonus points for not using Wordpress, I know it's tempting, but build something your self or use something more exotic. Any slob with a 56k can throw together a Wordpress site. Be sure to list it on your resume. I don't care how bad it is, fix it up, that's no excuse. If you have your name, or your email address or anything else I can surmise about you, I will find it. If I find it, I'll presume you were hiding it from me - and that doesn't come off well.

Have extracurricular code samples. Using Github or other similar service, and more importantly including a link on your resume not only shows a willingness to stand by your code, but show a versed knowledge of SCM. If you don't build things in your own time, you're not learning as much as you could be.

The best thing I can recommend is writing your own framework. All the best developers I know have at one time or another created their own framework. It is an amazing learning experience for you, and moreover it gives a huge amount of insight into the way you think, and the way you prefer to code

Resume Basics

Your resume exists to spark a conversation; it shouldn't be huge and it doesn't need every minute detail about every project you've worked on. If it goes over four pages odds are I'm not reading it all.

Leaving pieces vague helps me ask questions, which can help spark a conversation. Often the most important bits are in questions I didn't ask.

Be painfully consistent. Spelling/casing the same technology multiple ways reflects poorly on you. If you can't be consistent in your resume, what is to say you'd be consistent in your work?

Have the same name, address, email, and phone number on every document. Not only is inconsistency here really awkward when trying to contact you, but it helps me keep my notes on you organized. I may be interviewing several people that day and often these documents will be stacked on my desk between interviews.

Mention things you've built from the ground up. While framework knowledge is useful in places looking for an expert in a specific framework, ground up work demonstrates a broader depth of knowledge.

Don't send your resume as a .txt file. It's not cool.

Interview Basics

Always be on time, especially for a phone interview. Gathering people to make a call just for you to not pick up is infuriating.

When you are asked a question, elaborate! Go off on small tangents about why some piece worked or didn't. Smile devilishly and mention how something saved the day! That said, don't ramble, there is a fine balance. Don't spend minutes answering what should have been a several second answer.

Bring samples of work not included on your resumé. These are incredible talking points.

• Be ready to identify what specific pieces you worked on.
• Don't include very similar examples.

Ahead of time consider interesting challenges faced working on each, you will be asked. If the answer is none, it's probably not worth including.

Other Important Bits

Try to not be nervous. Easier said than done, but you don't need this job, or at least act like you don't. That's not to say be cocky - just that showing neediness as with a personal relationship, can scare people away.

Experience is far more important than education. Talk less about what you've done in school and more about what you've done professionally or even better on your own.

Don't drone on about wanting to work from home or needing quiet. Every developer is a bit of an introvert, but going on about it indicates problems working with others to come.

If you appreciate this tool, please consider a donation. It truly helps me out.

PayPal:

GitHub:

Ko-Fi:

Playing Minecraft, I like making circular things. I used a chart while I was building, but wanted to be able to make variable size ovals which is something I couldn't find a decent chart of or generator capable of, so I created this!

---

---

Go Fullscreen Show Old Generator Fork my source on Github!

]]> https://donatstudios.com/Simple-Single-Pass-Doubly-Linked-Flat-Tree-Building-Algorithm Fri, 06 Jan 2012 21:55:47 -0600 https://donatstudios.com/Simple-Single-Pass-Doubly-Linked-Flat-Tree-Building-Algorithm A hierarchy is a common structure, especially for structuring pages in web development. A problem with hierarchies is they often need to be flattened to be stored in a relational database, and then expanded again after being pulled out.

An Example Hierarchy

• Top level item
  • Sub Item
    • Sub Sub Item
      • Sub Sub Sub Item
  • Sibling of "Sub Item"
• Top level as well

A common and simple way to store them is with a table structure similar to the following.

id	parent_id	title
1	0	Top level item
2	0	Top level as well
3	1	Sub Item
4	1	Sibling of "Sub Item"
5	3	Sub Sub Item
6	6	Sub Sub Sub Item

Now there are several ways to process this into an array or other navigable structure. A painfully common and naïve way would be recursive queries ala:

function fetch_items($parent = 0) {
    $data = array();
    $qry = mysql_query("select * from `tablename` where parent_id = " . (int)$parent );
    while( $row = mysql_fetch_array($qry) ) {
        $data = array('data' => $row, 'children' => fetch_items($row['id']));
    }
    return $data;
}

The problem with this approach is the sheer number of queries this approach uses. Our simple example hierarchy would execute 6 queries, whereas more complicated hierarchies would execute many more.

A vastly superior approach is to pull all the data out with a single query, and then process it into the structure we need. There are several ways to do this, but I find the following DOM like structure this generates incredibly useful.

function fetch_tree() {
    $data = array();
    $qrt = mysql_query("select * from `tablename` order by parent_id asc");
    while( $row = mysql_fetch_array($qry) ) {
        $data[ $row['id'] ]['data'] = $row;
        $data[ $row['parent_id'] ]['children'][ $row['id'] ] =& $data[ $row['id'] ];
        $data[ $row['id'] ]['parent'] =& $data[ $row['parent_id'] ];
    }
    return $data;
}

Essentially, in a single pass through the data, this gets us an array of hierarchy elements, where rather than an element directly having its children, every entry has a pointer to its parent element as well as an array of pointers to its children. This is incredibly flexible, and as the tree is flattened it allows you to start at any point in the hierarchy by ID ala $data[<$id>], and navigate your way up or down. Viewing any elements children is as simple as $data[<$id>]['children'], and its parent as simple as $data[<$id>]['parent'].

I use this basic methodology a ton in my work, personal and professional (the comments and sitemap of this site are powered by it!) but the concept seems foreign when I try to explain it to people. I hope this explanation sparks some ideas for people.

]]> https://donatstudios.com/Damn-Simple-PHP-ASCII-Art-Generator Thu, 10 Nov 2011 13:18:43 -0600 https://donatstudios.com/Damn-Simple-PHP-ASCII-Art-Generator So yesterday I was bored and asked my friend what I should do. She replied "I don't know, draw ASCII art or something!" and sitting there with the terminal open I kind of wondered what it would take to write one of those image to ASCII scripts.

20 minutes later I had the following PHP Shell Script! It works by simply sampling the saturation of every Xth pixel and mapping that saturation to a character from an array. There is no nearest-neighboring or anything like that, just straight up sampling.

Huzzah!

And there you have it! A whopping 35 lines of code. There is a ton of tweaking you can do like changing the array of characters, or adjusting the $scale variable which is simply how wide of a sample it makes. Setting it to 1 for instance should sample every pixel on the Y and every other pixel on the X (so it scales properly). To the right is a sample output of the script.

Try it for yourself!

The script accepts local or remote image paths as the first argument. The following is a simple usage example you can use to get you started.

./ascii.php "https://donatstudios.com/images/site/JesseDonat.jpg"

It was a fun little project and I look forward to seeing if/what the internet comes up with for it. Fork me on Github, or post links in the comments! Happy hacking!

]]> https://donatstudios.com/Lion-Photobooth-Repair Wed, 27 Jul 2011 05:13:20 -0500 https://donatstudios.com/Lion-Photobooth-Repair

Update 2023-10-17

The script has been verified to work in macOS Sonoma.

When I switched from an older MacBook Pro to a MacBook Air the backup from my old MacBook Pro was corrupted and unmountable. However, I had preserved a backup of my Photo Booth Photos and wanted the Photo Booth application to recognize them. This is just a matter of creating a Recents.plist, an ordered list of the image files.

While adding these photos to the Recents.plist file Photo Booth uses seems straightforward enough, arranging hundreds of photos from different naming schemes across various OS versions manually is pretty tedious.

To fix this, I wrote a script that automatically creates a new plist file with the photos in the correct order. As a safety measure, the script backs up the existing Recents.plist as Recents.plist.bk.[time], allowing easy restoration if needed.

Usage

Please Note - NEW

This script is compatible with macOS versions up to macOS 14 (Sonoma). However, it relies on PHP. While PHP was bundled with macOS until macOS 12 (Monterey), subsequent versions require a separate PHP installation.

I recommend using Homebrew. You can find Homebrew installation instructions on their official website.

After setting up Homebrew, install PHP by entering the following command:

brew install php

To Begin With

• Ensure all photos and movies you wanted loaded into Photo Booth are located in your Pictures/Photo Booth Library/Pictures/ folder.
• Ensure the Photo Booth Application is closed to avoid overwriting our changes.

Simple Method

Added September 10, 2012
Open Terminal and run the following command.

curl -Ls https://gist.githubusercontent.com/donatj/1108691/raw/Photobooth_rebuild.sh.php | php

Git Method

Simply executing the following should both download it and set it executable.

git clone https://gist.github.com/1108691.git PhotoBoothRepair

Non-Git Method

• Save the script somewhere accessible to you via Terminal.
• Navigate in Terminal to the directory where you saved the shell script
• Set the file executable by running

  chmod +x Photobooth_rebuild.sh.php

• Execute the shell script

  ./Photobooth_rebuild.sh.php

• All Done - Fire up Photo Booth to see our results!

Known Limitations

• All Leopard format names will come first regardless of date simply for lack of anything to go on. They're just numbered and contain no date information.

]]> https://donatstudios.com/PhpED-Dark-Theme Thu, 07 Jul 2011 22:34:57 -0500 https://donatstudios.com/PhpED-Dark-Theme It would seem there is a fair deal of call for an updated version of my PhpED dark theme from colleagues as well as random folks on the internet who’ve seen screenshots of my PhpED installation. I previously had my Dark Theme for 5 linked on my Review of PhpED, but I figured I could have a page dedicated to it.

Screenshots

Download and Installation

The first step is to download the proper version.
Click here to download the PhpED 6 Version or here to download the PhpED 5 Version.

After you have downloaded the file you will need to replace your hl.cfg.

In Windows 7 or Vista this should be located in or around:

C:\Users\YOUR_USER_NAME\AppData\[Roaming / Local]\NuSphere\PhpED\config

Likewise in XP it should be located in something like:

C:\Documents and Settings\YOUR_USER_NAME\Application Data\NuSphere\PhpED\config

The final step is to simply change your background color to a dark color. The background color is oddly saved in a different configuration file than the highlighting. This can be found in Tools ⇒ Settings ⇒ Editor ⇒ Appearance

While you're in there you may want to tweak the settings some to your taste, but that is all there is to it.

]]> https://donatstudios.com/MPO-to-JPEG-Stereo Wed, 13 Apr 2011 23:41:33 -0500 https://donatstudios.com/MPO-to-JPEG-Stereo

Convert MPO to JPEG

This tool converts MPO files from devices like the Nintendo 3DS, Fujifilm FinePix Real 3D, or other 3D cameras that support MPO files, into more usable JPEGs (.jpg). It's useful for viewing and sharing 3D photos in a more commonly used format.

Unlike many other online tools that simply strip one of the eyes from the result, presenting you with a single image, this tool incorporates both the left and right photos, preserving the depth and detail of the original 3D images.

Additionally, it can convert MPO files into an anaglyph format, allowing the images to be viewed with standard bi-color 3D glasses.

Gallery

]]> https://donatstudios.com/WPF-VBNet-JumpList-Example Thu, 07 Apr 2011 05:00:31 -0500 https://donatstudios.com/WPF-VBNet-JumpList-Example Today I decided to spruce up a VB.Net application I wrote in college and still use almost daily. In the process of converting it from .Net 2 to .Net 4 I wanted to add a splash of Windows 7 goodness; there were some very simple operations I wanted to add to JumpLists.  The problem I ran into with all the examples I could find is that they either called entirely different applications (the Microsoft examples all called Notepad) or the example was in C#.  The application is in VB.Net and I intend to keep it that way, as it is my .Net language of choice.

The interesting problem with JumpLists is they are just shortcuts.  They call an executable with the specified arguments. Microsoft’s reasoning for this is that they are still available while the program is no longer running. The initial execution of an application placed in the taskbar registers the JumpLists, after which point they are cached by the system and continue to be available even when the application is no longer running. This potentially limits the usefulness of JumpLists as they do start a new instance of your application.  If anyone knows how to catch the arguments on a single instance application, I would be very interested to hear it.  The best I’ve come up with is simply passing messages to my current instance from a new instance and then terminating the second instance, but this is awkward to my eye.

XAML can be used to create JumpLists in the Application.xaml file, which many other examples will demonstrate – but the issue here is there is no way to get the path to the current executable via XAML, so you can only call other executables whom you know the full path to.

The answer then is to create our JumpLists dynamically.  The best place to do this is in your Application.xaml.vb file.  The New method will be used to instantiate and register the JumpLists, whereas the Startup event will be used to handle arguments passed to the new instance of our application.

Below is a very simple example of JumpLists not too far off from what I ended up going with in my own application. Feel free to leave me comments as to any improvements or changes you would make, or go ahead and fork the gist.

]]> https://donatstudios.com/PHP-Parser-HTTP_USER_AGENT Tue, 29 Mar 2011 22:51:09 -0500 https://donatstudios.com/PHP-Parser-HTTP_USER_AGENT

Demo

Requirements

• PHP >= 5.4.0

Download

The script is available over on GitHub or via Composer:

$ composer require 'donatj/phpuseragentparser'

Usage

The original procedural use is as simple as:

<?php

// v0 style global function - @deprecated
$uaInfo = parse_user_agent();
// or
// modern namespaced function
$uaInfo = donatj\UserAgent\parse_user_agent();

echo $uaInfo[donatj\UserAgent\PLATFORM] . PHP_EOL;
echo $uaInfo[donatj\UserAgent\BROWSER] . PHP_EOL;
echo $uaInfo[donatj\UserAgent\BROWSER_VERSION] . PHP_EOL;

The new object oriented wrapper form:

<?php

use donatj\UserAgent\UserAgentParser;

$parser = new UserAgentParser();

// object-oriented call
$ua = $parser->parse();
// or
// command style invocation
$ua = $parser();

echo $ua->platform() . PHP_EOL;
echo $ua->browser() . PHP_EOL;
echo $ua->browserVersion() . PHP_EOL;

Currently Detected Platforms

see: List of Detected Browsers

]]> https://donatstudios.com/RewriteRule_Generator Tue, 01 Mar 2011 01:58:44 -0600 https://donatstudios.com/RewriteRule_Generator

Find this useful?

Buy me a coffee!

---

PayPal:

GitHub:

Ko-Fi:

I find myself often needing to set up a large number of 301 redirects from an excel file of old to new url’s. Until this point, writing these had been a fairly exhausting process as you need to be certain to escape every character that could be picked up by the regular expression engine, or risk unintended consequences. On top of that, writing these rewrites to catch GET parameters in any order is complex to say the least. While working through a large list of 301s for a very picky server where Redirect 301 style redirects were not an option, I began setting up RewriteRule's by hand and decided there had to be a better way. Voila! My 301 RewriteRule Builder was born.

You can simply enter a list of tab or space delimited urls (copied from Excel works great) and it will spit out the generated RewriteRule’s for you. GET parameters on the old urls are broken up and built into multiple RewriteCond’s set up to work in any passed order.

I hope someone finds this useful. Any comments are welcome below, of feel free to fork me on github!

Please make sure you have RewriteEngine On somewhere near the top of your .htaccess, before you attempt any rewrites.

Need more help? The tool can only do so much, and sometimes you need human help. I offer .htaccess help as a paid service and have 10+ years of Apache experience. Send me a note on my contact page and we can discuss your options.

Fork my source on Github!

]]> https://donatstudios.com/Google_Killed_The_Internet Mon, 20 Dec 2010 04:35:10 -0600 https://donatstudios.com/Google_Killed_The_Internet Googles Page Speed “Optimizations” Make Learning Difficult

If you, as I, grew up during the web bubble we call the 90’s and were interested at all in web development my guess is that you probably didn’t learn most of your skills from a book or a class.  You probably would run across a site that had something cool on it and ponder  “Oh my, how does that work?” right click, view source, and after some digging were enlightened.  I’m certain this is how most of my generation learned their HTML/CSS/JavaScript skills, it is certainly how I did, and often how I continue though now I have tools to help me even further pick apart a page such as Firebug.

Where I take issue is CSS and more so JavaScript minifying. Google wants essentially all your CSS and JavaScript minified. This lessens the bandwidth Google has to use, but in the process makes the source complete illegible to a human being.

I personally, on this site, lose a fair deal of my Page Speed Score because “Minifying http://donatstudios.com/js/general.js could save 311B (23% reduction)”.   My general.js file is at the moment 1.3 kilobytes; I am losing points off my Page Speed score over 311 bytes simply because I want my source code to remain legible.  Can we please get a sanity check on this, Google.  311 bytes is not going to kill you.

SEO is Destroying the Spirit of the Internet

Back in the hay day of Geocities if you did a search you were likely to get a few if not mostly amateur pages in your results.  They weren’t usually well designed perse, but they were often very useful. I can recall of the top of my head a few instances where they saved the day.

In steps “Search Engine Optimization”. Corporations are gaga over paying people to dig through content, study their keywords, rewording things and saturating content with keywords (often at the cost of readability I might add).

Now let’s compare that to an amateur, who’s content is designed for human consumption rather than googles and now has no way to keep up in this arms race. There usually isn’t the money nor the desire to pump into SEO and content development.  They just want to provide some useful information to the public.  These kinds of sites are becoming increasingly hard to find, with corporate sites taking the lion’s share of the hits.

Frankly I believe a lot of great content is getting missed thanks to SEO.  The spirit of the internet years ago was driven around the fact that anyone could write something, and have it read by millions of people.  While this is still the case, I find it far less likely now than it has been in previous years.  I don’t think there is a solution; I just find the corporatization of the web a little disheartening.

]]> https://donatstudios.com/Grid_CSS_Versus_The_Web Tue, 19 Oct 2010 23:43:21 -0500 https://donatstudios.com/Grid_CSS_Versus_The_Web Grid Style Sheets Must Die

I have been aware of Grid Style Sheets for quite a while, but I long ago dismissed them as a fad like CSS Resets.  Recently though it came to my attention that OSCommerce 2.3 switched to the 960 Grid Style to replace its table based layout system.  I spend most of my development time working with OSCommerce and thought it would be good to understand this change so I could decide whether or not to integrate it into our fork.

My general impression playing with the latest nightly of OSCommerce 2.3 from Github though is that I don't like it – maybe I'm old school, but in my view Grid Style Sheets go against what we spent the previous decade fighting for, separation of our concerns… separating presentation and content.

You're Doing Internet Wrong

Let's talk about separation of presentation and content.

Ideally, content and layout information should be entirely separate.  Class names should be used to describe the data they contain, and nothing to do with the style the class will contain.  You should be able to completely rearrange / restructure a page simply by switching out a stylesheet.  The HTML should strive to be as purely a data delivery mechanism as possible, similar to that of an XML request, and the CSS is there to make it something a human can digest.

Using a Grid stylesheet – your class names purely apply to your specific layout.  Class names like "grid_1" or "container_12" are not indicative of content in the slightest, but rather where they are placed on the grid that is your page. Grid_1 will always be leftmost. Always. Might as well use <left>

Correction: Grid_1 will always be one twelfth of your page wide (on a twelve Column Grid) , Grid_2 two twelfths, et cetera. Nothing to do with describing the content contained.

Grid Style Sheets are Extremely Limiting

With a well-structured CSS layout, deciding to move your navigation from the left to the right is as simple as flicking a switch.  Usually after changing one to a few lines of CSS, and its switched. It cascades across your entire site and everything floats and adjusts.  Same goes for altering the width, or even the entire flow of your document. You can quite easily decide your left navigation is now a top navigation with no alteration to the content.

Contrast this with a grid style sheet – to reposition anything you will have to edit the HTML – change the class you call.  Restructuring a site is a major task. They are essentially tables done in CSS with all the drawbacks of tables with few of the benefits (cough valign cough).

Why Does this Matter?

Flexibility, sustainability, scalability – these are not just buzz words – these are important attributes of a well-developed site – these are things a grid layout does not deliver.

Let's look at this from a prebuilt application perspective. In OSCommerce 2.3 the left navigation is classed grid_4 which places it 4 gridlines out of 16, and the content area is push_4 which aptly pushes it over 4.  That defines not only where my navigation appears, but also its size.  Its inflexible without changing HTML, and quite honestly wouldn't work for the majority of carts I develop.

Final Thoughts

Grids, from a design perspective are not just useful, they're downright ideal.  Clear alignment of elements creating strong visual lines is something strived for.  Consistent padding and margins are all very important – and all of these are honorable virtues of a grid style sheet.

I definitely understand the draw and the time saved in the initial build of a site, but a meticulously well engineered style sheet supporting semantic CSS will in the long run pay back the initial investment many times over.

Post Script

While writing this, I've decided I really like the concept and aim of the grid style sheet – and would love it if it did not interfere with the semantic nature of CSS.  If I could simply do something like the following, I would be using it 10 minutes ago.

#leftnav {  
    imports: .grid2;
}

W3C – I'm looking your direction to see this happen.  Until then, I'm sticking with my old fashioned meticulously crafted CSS.

Update February 3, 2014:

While my position on grid CSS has not changed, there are now many many Sass, Less, Stylus etc CSS pre-parser grid frameworks which allow your CSS to still be semantic, while utilizing grid layouts. This is wonderful and I wholeheartedly endorse their use.

]]> https://donatstudios.com/lorem_ipsum_generator Wed, 22 Sep 2010 21:02:25 -0500 https://donatstudios.com/lorem_ipsum_generator

It strikes me now, many years later, that what I'd built was essentially a PHP markov chain

Several months ago I was wondering to myself if you could detect patterns in a list of words and then use those patterns to generate a new list of fake “words”. A kind of a flavored Lorem Ipsum generator, where I could change the feel of the sentences by switching out the data set.  I’m sure thousands of developers have done this before me – but it was a fun thought experiment.

I threw together a little script in PHP to test the idea, and the results were kind of interesting so I figured I'd throw it on my site - Anyway, here are some examples in action – you can hit reload for a random ipsum!  The code and a sample dataset are below.

Faux English Demo

Faux Latin-y English Demo

Faux Japanese Demo

You need to supply a set of words for it to base the pattern off - here is my sample dataset.
]]> https://donatstudios.com/FastPatch Fri, 27 Aug 2010 21:59:26 -0500 https://donatstudios.com/FastPatch In my work, I deal with a lot of very similar codebases - and often if I fix something in one project I'll want to fix it in many others. For a long time this has meant popping open Beyond Compare, which works, but isn't the simplest solution.

I use git on my projects - but they're not similar enough just to be branches. Recently I came up with a way to patch one or more commits from a project to another easily. Here is a simple shell script I wrote to handle the task.

Simply put, you can use it either by

$ fast-patch.sh /z/my_project

to patch a project with the latest commit from the project in the working directory or with an optional parameter, specify a number of commits, eg

$ fast-patch.sh /z/my_project 5

Also, it will generate a patch log to ~/fastPatch.log which I generally use directly for my commit message by way of

$ git commit -F ~/fastPatch.log

Any comments on how I could improve it, or questions are quite welcome.

]]> https://donatstudios.com/Balanced_Take_on_Gzip Tue, 17 Aug 2010 00:30:31 -0500 https://donatstudios.com/Balanced_Take_on_Gzip With Google recently starting to involve load time in search rankings, there has been a lot of talk about GZIP.  Google Webmaster Tools exclaims "Compressing the following resources with gzip could reduce their transfer size by XKB:". Many people listen to this, and go out looking for a quick fix without realizing that poor GZIP implementation has a strongly negative effect on the overall user experience.

There are two main ways in PHP to go about implementing GZIP. The first is ob_gzhandler.  Basically all that is really involved in setting this up is adding ob_start("ob_gzhandler"); anywhere before headers are sent, and it will blindly handle this for you.

The second is zlib.output_compression which while entirely transparent and Zends states that using zlib.output_compression is preferred over ob_gzhandler().

This method can have a huge downside for users.  It can for instance greatly exacerbate the sluggishness of an already slow page or even site. Right now your probably saying to yourself "What? I thought GZIP made everything faster. This guy is crazy.". Allow me to explain.

Let’s say that you have a site with no all encompasing output buffer.  As PHP works its way through your script it occasionally flushes out its internal buffer to apache, which in turn flushes it out to the browser, allowing the browser to begin rendering the incomplete document. Also, let’s say as an example you have a hold up slowing down your page in the footer of your site. The user would already have the rest of the content and could begin reading and enjoying the site.

Now let’s compare to a site GZIPed with an output buffer callback, e.g. ob_gzhandler. After the output buffer is started, your code begins to execute.  Everything echoed is held in the buffer until the page is completely done and the output buffer closes, at which point it is GZIPed and flushed out to the user.  The user recieves less bits, but that hold up in the footer we spoke of before halts up the entire rendering process, and the user cannot see anything until the footer completes.

For your viewing pleasure, a demonstration:

Side by Side Comparison

Non-GZIPed	GZIPed

It has come to my attention that Webkit browsers wait for a full result in an iframe to begin rendering - so this example will not work in Safari/Chrome

Take Note

I am not stating that all GZIP is bad. To the contrary, GZIP can be very beneficial and I absolutely sympathize with google wanting to cut down on their bandwidth. If you have Firebug or something similar you can note that the request for the Non-GZIPed example is a whopping 100 kilobytes, whereas the GZIPed examples request is a measily 323 bytes! The reason for this is I did not have enough content to get Apache to flush so to get around this I added the following line.

echo str_repeat(' ', 100000); //This creates 100kb worth of data, enough to trigger my copy of Apache to flush;

Not only does this trick cause Apache to flush, but it exagerates the positive effect of GZIP. Simple patterns, or in this case large amounts of whitespace, will compress quite wondefully.

Uncompressed static files, JavaScript and CSS being two of the best examples compress quite well. The best way to do this is on an Apache server is with

<IfModule mod_deflate.c>

<FilesMatch "\.(js|css)$">

SetOutputFilter DEFLATE

</FilesMatch>

</IfModule>

Looking for a Solution

I've posted a number of questions to Stack Overflow [1] [2] researching this writing and actually found very limited help.

It appears murky whether or not the protocol even supports the functionality of chunking GZIP results into separately compressed blocks, as would be required to decompress portions before the page is done – you can chunk a complete GZIP response, but the advantage of this is limited.

Lastly I have seen “promises” of ob_gzhandler caching the results leading to a speed boost on a number of blogs – this is purely fiction and not to be trusted.

Moral of the Story

Understand what you’re doing before you do it.  Blindly using output buffering for GZIP can have adverse effects on your site. 

]]> https://donatstudios.com/SimpleCalendar Thu, 01 Jul 2010 01:17:11 -0500 https://donatstudios.com/SimpleCalendar SimpleCalendar is a very simple PHP calendar class written to be easy to use and flexible.

It includes basic event logic and is made to be modified to your particular needs.

Sample Usage

<?php
require_once('SimpleCalendar.php');
$calendar = new donatj\SimpleCalendar();

$calendar->setStartOfWeek('Sunday');

$calendar->addDailyHtml( 'Sample Event', 'today', 'tomorrow' );
$calendar->show(true);

Example Calendar

Click here for the Github repository.

Recent Activity

Nov. 24, 2025

GitHub (web-flow)

Merge pull request #32 from donatj/donatj-patch-2

Add PHP version 8.5 to CI workflow

---

Add PHP version 8.5 to CI workflow

---

Nov. 21, 2025

Bump actions/checkout from 5 to 6 (#31)

Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
dependency-version: '6'
dependency-type: direct:production
update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---

Aug. 12, 2025

Bump actions/checkout from 4 to 5 (#30)

Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
dependency-version: '5'
dependency-type: direct:production
update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---

Nov. 24, 2024

Merge pull request #29 from donatj/donatj-patch-2

Test in PHP 8.4

---

Test in PHP 8.4

---

Nov. 27, 2023

Merge pull request #26 from donatj/php-version/8-3--users-jdonat-projects-donatj-simplecalendar--github-workflows-ci-yml

Update PHP to 8.3

---

Jesse Donat (donatj)

Update PHP to 8.3

---

Nov. 2, 2023

Replace travis badge with GitHub Actions

---

GitHub (web-flow)

Merge pull request #25 from donatj/phpstan

Pretty major cleanup

---

Jesse Donat (donatj)

Use Makefile

---

Adds helpful Makefile

---

Update autoloader syntax

---

Use correctly compatible versions

---

Updates README

---

Minor fixup

---

Drop pre-7.2’s

---

Run fixer

---

Add phpcs and php-cs-fixer

---

Fixup

---

Add phpstan.neon

---

Cleans up composer.json some

---

Correct backwards annotation

---

Remove unnecessary if

---

Min PHP 7.2

---

Fix http url

---

Sep. 5, 2023

Bump actions/checkout from 3 to 4

Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
dependency-type: direct:production
update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

---

Jan. 11, 2023

Updates docs

---

GitHub (web-flow)

Merge pull request #22 from donatj/8_2

Test in PHP 8.2

---

Jesse Donat (donatj)

Resolve PHP 8.1+ deprecation notice

---

Build	Date	Message
v0.6.0 Download	Apr. 16, 2014	Simplified shrimp2ts work
v0.6.1 Download	Apr. 16, 2014	gitignore updates
v0.6.2 Download	Jun. 7, 2017	Merge branch 'keesiemeijer-master'
v0.7.0 Download	Jan. 11, 2023	Updates docs
v0.8.0 Download	Nov. 2, 2023	Replace travis badge with GitHub Actions

]]> https://donatstudios.com/JDZoom Tue, 08 Jun 2010 03:18:35 -0500 https://donatstudios.com/JDZoom

Recent Activity

Feb. 4, 2019

Jesse Donat (donatj)

Readme/LICENSE fixes

---

Jan. 22, 2014

Merge pull request #1 from bitdeli-chef/master

Add a Bitdeli Badge to README

---

Bitdeli Chef (bitdeli-chef)

Add a Bitdeli badge to README

---

Oct. 25, 2010

Jesse Donat (donatj)

Fixed an issue with cross domain images

---

Oct. 8, 2010

Fixed a regression where selector would only accept strings

---

Oct. 7, 2010

Restructured Repo per MooTools Forge requirements

---

Updated README.md

---

Added Options:

Magnified Position Option Added
options: fixed/float

Cancel Click Option Added
boolean, prevents default on an images anchor

Misc Code Cleanup...

Fixed "endless.com" style added

index.html demo's updated - shows many different
options and settings in action.

---

Added escaping to css url() calls to prevent problems with images with
spaces in file names.

---

Large Rework of Functionality
- Chrome compatability added
- Zoom box to magnified result math corrected
- Image preloaded
- Div now hovers over image rather than replacing
- Selector option added allowing an element, array of elements,
or a string selector to be passed.
- Internet Explorer Compatability fixes.

---

Jun. 7, 2010

README.md and .gitignore added

---

Initial Commit

---

JDZoom is a MooTools powered hover zoom / magnifier similarly styled to the one featured on Endless.com.

It is supported by IE6+, Firefox 2.5+, Safari 2+ and Chrome

It does not interfere with click though, such it can be used in conjunction with almost any lightbox. In fact by altering the selector when you instantiate it, most lightboxes can be extended with JDZoom without touching the HTML at all. See the advanced demonstration for more details.

JDZoom requires MooTools, and is Compatible with 1.2.5+ and 1.3+

It does not require any components of MooTools More.

Click here for the Github repository.

Demonstration

View Advanced Demonstration

Example Source

<html>
<head>
	<link href="https://donatstudios.com/<a href="https://github.com/donatj/jdzoom/blob/master/Demo/css/jdzoom.css">css/jdzoom.css" rel="stylesheet" type="text/css" />
	<script src="js/mootools.js" type="text/javascript"></script>
	<script src="js/jdzoom.js" type="text/javascript"></script>
	<script type="text/javascript">
	window.addEvent('load',function() { 
		var jdz = new JDZoom();
	});
	</script>
</head>
<body>
	<a rel="jdzoom" href="https://donatstudios.com/<a href="https://donatstudios.com/assets/18/Demo/images/venus.jpg" target="_blank">images/venus.jpg"><img src="https://donatstudios.com/<a href="https://donatstudios.com/assets/18/Demo/images/venus_thumb.jpg" target="_blank">images/venus_thumb.jpg" /></a>
</body>
</html>

Build	Date	Message
v0.1 Download	Jul. 27, 2010	Initial Alpha
v0.9 Download	Oct. 7, 2010	version 0.9 - nearing release
v0.9.1 Download	Oct. 7, 2010	version 0.9.1

]]> https://donatstudios.com/XML_Excel_Exporter Mon, 07 Jun 2010 22:54:58 -0500 https://donatstudios.com/XML_Excel_Exporter Update 2017: The functionality of "XML Excel Exporter" is now part of the Quorum Exporter project.

Recent Activity

Oct. 15, 2010

XSpreadsheet dies after outputting the spreadsheet now, preventing corruption

---

Feb. 18, 2010

Repository Restructured
-Working Corpus now in Source Folder
-`Scripts` folder added, includes database dump batch, and README.md HTML viewer
-`Local` folder added, for testing use with local database connections, etc

Documentation
-Finally! Created an initial README.md
-TODO moved from includes folder into root

---

This simple class allows you via PHP to export an XML Spreadsheet compatible with all versions of Excel 2003 and up.

The files generated will work with versions of Office 2003 and up. It is considered production safe, and is currently in use in a fair number of production sites.

Unlike CSV or HTML Table exports, you can have multiple worksheets of data! A sheets data can be added to the spreadsheet either by a MySQL query resource or as a 2 dimensional array.

Known bugs:

• Office 2007/2010 bark about XLS file extensions despite refusing to open XML file extensions downloaded from the internet. (Anyone know a solution?)
• DOMDocument incorrectly converts/removes non-printable/low range characters (eg: &#10;).

Limitations:

• File Format is not supported by:
  • Open Office
  • Google Documents
  • Microsoft Office Live

MySQL Example

require('includes/classes/XSpreadsheet.php');
$spread = new XSpreadsheet($fname)
$spread->AddWorksheet('Products', mysql_query("Select * From products"))
    ->AddWorksheet('Categories', mysql_query("Select * From categories"))
    ->Generate()->Send();

Array Example

require('includes/classes/XSpreadsheet.php');
$data = array(
    array( 'Column 1', 'Column 2', 'Column 3' ),
    array( 1, 2, 3 ),
);

$spread = new XSpreadsheet($fname)
$spread->AddWorksheet('Awesome Sheet', $data )
    ->Generate()->Send();

Click Here to Download]]> https://donatstudios.com/No_App_P1_PhpED Thu, 07 Jan 2010 20:53:36 -0600 https://donatstudios.com/No_App_P1_PhpED “There Isn't an App For That” - Windows Apps I haven't been able to replace in Mac OS X

For a very long time I was a Windows guy, I was once a DOS guy. I switched nearly exclusively to Mac for my home use about 4 years ago. I’ve been very happy with it. Its stable, well engineered, and - my favorite part of all - consistent. Yet, in this time there remain a few applications I cannot seem to replace.

Nusphere PhpED

I am a PHP Developer by trade, and PhpED for my money is by far the best development environment for the serious PHP developer. When I do develop on the Mac platform it is usually split between two applications – Coda and TextMate. They are fantastic text editors, don’t misunderstand, but when you are developing highly classed applications a strong IDE becomes highly desirable. PhpED’s IntelliSense style PHP auto completion is on par with and in some ways surpasses Visual Studios.

Applications like Dreamweaver, Coda, TextMate and Espresso by comparison provide very basic auto completion. Zend Studio, Aptana Studio and other Eclipse based solutions have varying qualities of auto completion, but are awkward due to Eclipses generality and slow and visually out of water due to being built on Java. Komodo comes the closest, but can be very slow at times. Weighing in at $299 its pricetag is also heavy.

To give you in an example, in my CorpusPHP Framework the user class is built specifically to be extended for the sites particular needs. The login class therefore has a member instance of whatever type of object extending user you might have. This member is dynamically set as an instance and therefore type cannot be determined by most editors and auto completion will not work.  PhpED on the other hand has a solution. This brings me to my favorite PhpED feature - deep phpDocumentor integration.

class Login {

    /**
    * Object that holds the user object, brought over from the session
    * @var User
    */
    public $user;

    … … …

}

The above code snippet shows the user member of the login class. It can be any type of user, for instance admin or limited. It will though be inherently be something that extends the user object.  Using phpDocumentor syntax here we imply that the type is a user and since PhpED is smart, it auto completes based on this.  If you are unfamiliar with phpDocumentor syntax you should read up on it, and begin marking your PHP up in this standardized syntax.

PhpED takes this integration further and will show descriptions of methods and functions as well as their parameters and return values based on their inline phpDocumentor documentation.

While auto complete is the PhpED killer feature in my book, it has many other excellent features.

 Its internal debugger  is second to none, and requires no setup unlike other debuggers. It installs its own copies of Apache and PHP 4 & PHP 5.2 or 5.3 which it uses to debug. The debugger can output to an internal version of IE or Mozilla or any external browser. Its quite configurable, everything you would expect like ability to set $_GET, $_POST, etc variables, break points, runtime analysis.

Some smaller features I personally enjoy are for instance:

• To-Do list shows you all the todo’s throughout your project.
• Continually improving, high quality CSS/HTML auto completion.
• The ability to set a default character set on a per-project basis
• Non-Ascii range characters show up dark blue in variable and function names, making them obvious – which is particularly helpful when working on code from a Cyrillic coder whereas Α (Alpha) and A are visually identical but entirely different characters.
• Great support – Support will work with you to fix a problem.

Update 7/7/2011

The dark PhpED Theme I use is now available here for PhpED 5 and 6.

]]> https://donatstudios.com/CorpusPHP Wed, 02 Dec 2009 21:21:04 -0600 https://donatstudios.com/CorpusPHP

Recent Activity

Mar. 11, 2019

GitHub (web-flow)

Update README.md

---

Apr. 14, 2015

Jesse Donat (donatj)

Correct year and formatting

---

Create LICENSE.md

---

Jan. 7, 2014

Merge pull request #9 from bitdeli-chef/master

Add a Bitdeli Badge to README

---

Bitdeli Chef (bitdeli-chef)

Add a Bitdeli badge to README

---

Jul. 15, 2013

Jesse Donat (donatj)

Better constant

---

May. 16, 2013

Cache class made to serialize

---

Apr. 26, 2013

Corrections to the Github api modules pulled in

---

Mar. 5, 2013

See doesn't die. Drop submits an error code.

---

Oct. 15, 2012

Update Source/corpus/modules/TwitterFeed.php

Change in the twitter API url

---

Sep. 5, 2012

Update TODO

---

Aug. 17, 2012

Basic SASS/Compass support added

---

Jul. 11, 2012

Merge branch 'master' of github.com:donatj/CorpusPHP

Conflicts:
  corpus.sql

---

Last.fm module allowed to float on its own

---

Comments upgraded to DonatStudios hierarchical system

---

Tag cloud improvements from Donat Studios

---

supplementary column added for supplementary JSON encoded data about a page

---

db_dump shell script added to scripts

---

SQL dump upgraded to 5.5 looking file

---

Jul. 10, 2012

Brfeadcrumb code given itemprops

---

Comment fix

---

More useful MySQL errors

---

Jul. 9, 2012

headers_list_assoc function allowing headers_list result a an associative array added

---

Jun. 5, 2012

Update master

---

Jun. 1, 2012

old template => layout correction

---

May. 15, 2012

HTML5 cleanup-y stuff

---

Mediabox advance removed

---

Basic namespace support roughed in to the autoloader

---

Made to throw errors by default in silly environments

---

Merge branch 'master' of github.com:donatj/CorpusPHP

---

CorpusPHP is setting out to be a world class PHP Framework. The aim is simple, to provide encapsulation, modularization, and helpful tools, all without changing the way you write PHP.

Project Information, documentation, and a timeline for the first release available availble soon

Nightly's available at GitHub, or download the latest milestone below.

Build	Date	Message
v0.9 Download	Feb. 18, 2010	Major steps made in maturity, approaching release candidate
v0.92 Download	May. 14, 2010	Version 0.92
v0.98 Download	Sep. 7, 2010	Version 0.98
v0.981 Download	Dec. 21, 2010	Optional $seperator paramter added to the breadcrumb generator.
v0.99 Download	May. 29, 2011	version 0.99

]]> https://donatstudios.com/Donat_Studios_Launches Tue, 27 Oct 2009 02:45:28 -0500 https://donatstudios.com/Donat_Studios_Launches After much ado, Donat Studios is finally live!

There are a few things left to complete. Eventually setting up the documentation and public git repository for CorpusPHP will certainly take up a fair amount of time. I'm still not quite sure what to put in my portfolio, I was thinking of for the time being just throwing a bunch of my sketches in there.

My reasoning and goals behind this site are slightly different than my other sites. More than anything I wanted a place where I could rant about everything web related. I have strong opinions on many topics, and where Oasisband.net is good for movies and politics, and PHPStandards is pretty much just raw anger directed at Zend, and my live journal is for my personal life. Donat Studios on the other hand will allow me to express my thoughts on web development, databases, html, php, ajax, design, basically everything I do at work.

Where does the name "Donat Studios" come from? I was getting some photos printed by a great place in St. Paul, "White House Custom Color" and they needed to know the name of my studio. I didn't have a studio, so I quickly made up "Donat Studios". Later, I realized I liked the name, and it stuck.

Here's wishing my newest domain luck!

Building this site I wanted to bring in images from my Flickr account as you may have noticed. Building the module in PHP, I basically ran file_get_contents on the feeds url and passed that directly into json_decode. It didn't work, the result was null. It had failed to decode. As a test I echoed the result of the file_get_contents to see what was up; it was succeeding at retrieving the feed. At a glance it looked like JSON, yet it was failing to parse. I knew something was up.

Here is an example result set:

jsonFlickrFeed({
  "title":"Uploads from donatj",
  "link":"http://www.flickr.com/photos/donatj/",
  "description":"",
  "modified":"2009-05-31T08:24:46Z",
  "generator":"http://www.flickr.com/",
  "items":[
    {
 "title":"STP81353",
 "link":"http://www.flickr.com/photos/donatj/3581174864/",
 "media":{
   "m":"http://farm4.static.flickr.com/3568/3581174864_be1a44764e_m.jpg"
      },
      "date_taken":"2009-03-30T19:10:34-08:00",
      "description":"<p><a href=\"http://www.flickr.com/people/donatj/\">donatj<\/a> posted a photo:<\/p> <p><a href=\"http://www.flickr.com/
photos/donatj/3581174864/\" title=\"STP81353\"><img src=\"http://farm4.static.flickr.com/3568/3581174864_be1a44764e_m.jpg\" width=\"180\" height=\"240\" alt=\"S
TP81353\" /><\/a><\/p> ",
      "published":"2009-05-31T08:24:46Z",
      "author":"nobody@flickr.com (donatj)",
      "author_id":"30444376@N08",
      "tags":""
    }
  ]
})

On further examination I noticed, as you should have noted, the data is wrapped in a function. My first thought was that this violates the JSON Specification, so I immediately tweeted about how silly it was for Flickr to have an invalid JSON feed when their parent company Yahoo employs Douglas Crockford, the creator of JSON. I soon received a response from a colleague informing me that this was JSONP. He explained what it was, and how to use it in jQuery. I had not heard of JSONP.

It basically comes down to this: XMLHttpRequest will not, in order to protect against XSS, work across domains. Therefore, JSON cannot be directly retrieved nor parsed via JavaScript across domains.

This is where JSONP comes in. JSONP is essentially a regular JSON result set wrapped in a function. Rather than using XMLHttpRequest to retrieve the document, you dynamically attach a script tag to the page pointing the the JSONP document, which evokes a function call passing your JSON object to whatever function the JSONP is set to pass the data to, forgoing the need to eval anything.

I have a number of criticisms of this approach.

• First and foremost, JSONP is at its very heart remote code execution. You must have absolute faith not only in the intentions of the site you are bringing the feed in from, but the security of said site as well, because the code is in fact being executed by the browser with no chance for putting regular expressions or other such safety mechanisms in place. This makes your site and visitors wide open to any and all malicious code a hacker may plant in the feed.
• JSONP is a language specific data type. What I mean by this is that JSONP is designed for JavaScript, and even then strictly in the context of a web browser. There is no need, none, for JSONP anywhere else or in any other language (PHP, ASP.Net, Ruby on Rails). Making a feed in a language specific format is inherently in bad form.
• Adding to the last point, other languages currently cannot naturally parse JSONP. As I ran into in PHP, it is not valid JSON and json_decode will not handle it. My colleagues answer was to write a regular expression to remove the function, but this is at best a hack.
• Mixing function and data goes in the face of separation of concerns. You’ve got logic in your data, you’ve got data in your logic. Bad form, good sir, bad form.

JSONP is in many cases an unsafe practice, and more over its just evil. It is a necessary evil in some cases, but evil none the less.

There is at least one safe alternative to JSONP. The simplest solution provided you have access to server side code is to download, sanitize, optionally cache, and finally pass locally any remote JSON to the client rather than having the clients machine directly load the feed. Yes, this is added overhead on your side, bur for much added security.

If nothing else, it is up to you to weigh the risks and benefits before blindly using JSONP.