Werd I/O

We've got some exciting new things in store for 2016 that solve real problems for both higher and corporate education. We'll discuss this in a future post on the Known blog. First, though, I wanted to take a step back and explain the technical decisions we made for Known.

What is Known?

Known is an open source web platform that allows groups and individuals to publish in a group with a variety of media. You can choose who can see the content you publish, as well as where you reach your audience: you can syndicate your content to services like Twitter, Facebook, SoundCloud, Flickr, LinkedIn and more.

It's also an open platform designed to be extended:

• Every content type is provided by a plugin, so any organization can add new kinds of content. (For example, we don't provide video out of the box - but you could.)
• Every syndication service can also be extended, so while we provide plugins for social media, a university could extend Known to allow students to submit their work to their Learning Management System. Internally in the company, we'll often create Known sites for particular projects and then syndicate our posts to Slack.
• Known supports themes.
• Plugins can also provide Single Sign On (we provide LTI and LDAP to our enterprise customers, but for example, KQED uses SSO to link Known to WordPress accounts).

Known works for a single user - my website runs on it - or five thousand. It's up to you.

Did I mention it's fully responsive, meaning it works just as well on your smartphone as it does on your laptop? Or that every page is an API endpoint?

Install anywhere, extend easily

A key goal for Known is the ability to install it virtually anywhere.

Installing self-hosted web software is, unfortunately, not as easy as installing an app on your iPhone or your laptop. However, it doesn't need to be a developer-centric process.

Shared web hosts are immensely popular, and abstract away a lot of the really technical work involved in maintaining a server. You can often select an application to install from a directory of available projects, answer a few questions, and be ready to go in a couple of minutes. At its hardest, you can upload some files via FTP. You never have to drop to a command line and run Linux commands - and indeed, often you can't.

We wanted to be compatible with these hosts (our web hosting sponsor is DreamHost), as well as power users who have deeper technical control over their servers. That implied a number of requirements:

• The software language needs to be compatible with a large number of servers
• Users need to be able to install the software without command line tools
• Knowledge of version control systems like git, or managers like GitHub, shouldn't be required
• Use of a package manager like npm or composer shouldn't be required

It turns out that the most widely-supported language on shared hosts is PHP.

PHP has received not a small amount of scorn in developer circles over the last decade, and a lot of it is fairly earned. But the truth is that modern versions - particularly 5.4 and above - have consistent interfaces, and modern language features like namespaces and closures that bring it closer in line with more cutting-edge languages. The PHP style recommendations produced by the Framework Interop Group and popularized by PHP The Right Way have done a lot to standardize PHP code.

In fact, PSR-4, which defines a template for class namespaces and a way for objects to be autoloaded on demand, turns out to be useful. Every plugin in Known uses this standard for autoloading.

The only question is PHP version: not every host supports these features. In fact, while it turns out that 98.8% of PHP hosts support version 5 or above, 34.3% of these are on version 5.3. We expect this number to shrink over time, and consider it acceptable to be supported by the remaining 65% of web hosts. The syntactic features you gain, like closures, are worth it.

To support virtual URLs, we initially required the Apache web server (which is still the leader overall on the web). However, a number of community members have created open source configurations for nginx.

The data model

I don't think it's acceptable for plugins to create and maintain their own database tables. For one thing, you may wish to prevent Known from having database modification access permissions. For another, this means that every plugin is a potential database security risk or performance drain.

Instead, from the beginning I wanted plugins to access the database via an abstracted interface, and never have to worry about the schema. At the same time, I wanted plugins to be able to store any data they needed to function, in a way that made sense in the context of that plugin.

The first versions of Known used a NoSQL database, MongoDB, as its sole data store. This worked well for development, but it quickly became apparent that shared hosting would not support this as a data layer. In interview after interview, users said they wanted to run Known on hosts like Reclaim Hosting and Nearly Free Speech. In fact, many shared hosts support MySQL - and that's it. This left us with a challenge: could we provide a schemaless database layer while providing full support for MySQL?

Kevin Marks provided the answer: a balanced schema developed by FriendFeed back in the days before NoSQL databases became commonplace. We created a highly-indexed metadata table, which is purely used for searching for objects, and then stored the complete objects in JSON in an object database. All of this is provided by a seamless database layer called the Data Concierge, that abstracted many of the functions provided by the MongoDB PHP extension.

A side effect of this abstraction is that more databases could be added easily. Today, as well as MongoDB and MySQL, Known supports SQLite and Postgres.

Distributed social networking and uncool URIs

One of the core original visions for Known was that data could be distributed. A user on site A could participate in a community on site B. Imagine creating a group for a project across two companies, and then allowing users from a second company to join and collaborate without re-registering! There are lots of real-world possibilities for distributed social networking.

To prepare for this, we decided that every object would have a URI as its definitive UUID. The idea was that you could access any resource by its UUID anywhere on the web, and as long as the request was properly signed, you'd be able to access it as if it was locally stored. In the end, I consider this a core mistake, but one that is hard to move away from.

Tim Berners-Lee famously said that "cool URIs don't change". Unfortunately, in the real world, URIs change all the time - and there's no way to require that they don't.

• Domain names expire
• People lose control over their domain names (eg a subdomain at a university)
• People choose to move sites into subdirectories

Imploring people to strongly consider their website layouts, as the W3C does, is not helpful for individuals who just want to run a site. The web is not set in stone; websites change, and URIs should be treated as volatile in any internal data model. 

As it stands, Known contains a number of protections that allow it to be moved to different domains or directory locations, so users don't notice a difference. It's not a technical decision I'm proud of - but it may yet come into its own. We already use the indie web technologies for some distributed social networking, and it's an idea that I'm convinced will transform the web.

The front end

Creating a native mobile app for a platform that can be infinitely extended is difficult. Instead, we created a fully responsive, touch-friendly interface.

Known separates model, view and controller, and any page can be viewed with a different template. For example, here's my website using a JSON template, and here's a Star Wars crawl. Any plugin or theme can override any template element, so I could write a plugin that changes out the WYSIWYG editor (we use TinyMCE), or that displays avatar images as 3D spheres (if I really wanted to). I could write a template to display Known sites using a virtual reality browser - and someone really should!

For the default template, we chose Bootstrap and jQuery. The former provides a solid, responsive UI that can be extended easily (and which removed the need to develop it from scratch). The latter provides a powerful, performant way to query elements on the page. Not only did this combination let us get up and running quickly, but plugin authors could use them to create simple, grid-based user interfaces that would be in line with the platform as a whole.

For glyphs like social media logos, we use FontAwesome. The latest version contains 605 different user interface icons, is well tested, has a good community, and a compatible open source license. All of these things made it perfect for our use - and, again, making features available to plugin authors.

Every page is HTML5, CSS3. Content is encoded using microformats, allowing software to read and extract meaning from our human interfaces. This forms the basis of important decentralized social web protocols like those used by the indie web community.

Over time, we've learned that we do need to support a mobile app. The mobile web has evolved to be decent for consumption, but there are obvious missing pieces for producing content on a mobile over the web.

For example: it's difficult to upload media. Resizing camera JPEGs in front-end Javascript on a mobile device is not a reliable process. The web audio API produces WAV files, rather than MP3s, which are uncompressed and potentially large. We could resample these on the server side using something like ffmpeg, but it's not reasonable to expect a shared host to support media encoding - and nor is it reasonable to expect users to link up to a third-party media encoder like Zencoder. Worse: we found that the web audio API actually crashed many mobile browsers!

This problem is compounded by video uploads. Video files are huge, and there's no way to compress them in a browser. Backround uploads are hugely tricky, and resuming failed uploads is also hard. That's even before they've reached the server - and when files can be as large as 1GB per minute of footage, both storage and encoding is hard.

For the mobile web to effectively compete with apps, it needs to support the content composition experiences that native apps have been using for years. If we want people to build websites, the web needs to support building, across devices. It's a frustration, and an ongoing problem.

Moving on

Our PHP-based infrastructure and need to support shared hosts means that some features are much harder to produce. The truth is that technologies like websockets (useful for performant real-time user interfaces) are hard for non-developers to self-host. New web platform features like web workers show enormous promise, but require secure connections - and even with empowering projects like Let's Encrypt, setting up secure sites is still too complicated for most people.

The good news is that some progressive enhancement is possible: companion services that provide extra capabilities to hosted software. It's also true that hosts are evolving, and our friends at DreamHost and Reclaim Hosting are thinking hard about the future of the space. 

I'm proud of the platform we've created - it's one we use every day, and I'm delighted to see people posting on their own servers all over the world. We've got big plans for the Known open source project this year, and we're looking forward to sharing them with you, in conjunction with something new that we'll tell you about soon.

It's going to be a great year.

This afternoon, Greg Wester tweeted this screen grab:

Your eyes are not deceiving you. That's a 420 square foot studio apartment - not even a one bedroom - for $3,050 a month. That's a base cost: your bills, and most likely parking, are extra.

There are only two kinds of people who can afford this:

• Workers on a substantial salary (realistically $100K+, which would still leave you with a bi-weekly check of a little under $2500 after taxes, meaning rent would be well over 50% of your monthly cost)
• The independently wealthy

Let's leave aside the obvious social inclusion issues at play here, and the effect this has on diversity in the city. Let's ignore that this is killing the artistic temperament of the city and turning it into a primarily financial center like any other. Let's pass over the inevitable effect this will have on the city when these high-value residents start to ebb away. Let's pretend not to see the rising homelessness problem. Not because those are unimportant issues - they're vital to the future of San Francisco - but because it's harmful to the ecosystem that helped create this situation to begin with.

If only rich people can afford to live in San Francisco, it is impossible to really innovate. All the creative energy is being driven out. There's no way for ordinary people - people who haven't made it yet - to experiment. Everyone is either on salary or has raised money from institutional investors with a proven business model.

So the gentrification cycle turns:

Deindustrialization creates low-rent vacancies in industrial districts; artists are drawn to these districts by the depressed rents and spacious "lofts"; the district becomes a hub of avant garde creativity, generating media attention and foot traffic, both of which create a "buzz" around the neighborhood; shops and restaurants are drawn to the area to cater to the increased foot traffic and capitalize on the "buzz;" the introduction of these shops and restaurants in turn induces more foot traffic, more media attention, and more "buzz;" eventually national chain stores see the area as ripe for investment and begin to move in; finally, of course, each of these trends causes rents to escalate until, with the arrival of deep-pocketed chain stores, the very artists who made the district trendy are priced out.  The district ends up as nothing more than a high-end outdoor shopping mall with little street "cred," and the artists relocate to a new low-rent industrial area, triggering the process all over again.

We've seen this process start to rapidly transform Oakland:

Oakland neighborhoods that are experiencing "advanced gentrification," according to the study, include Lower Bottoms, Old Oakland, and Northgate/Koreatown. The researchers define "advanced gentrification" as areas that have experienced significant demographic changes and high levels of real estate investment. Those areas are also very vulnerable to gentrification due to their locations near transit, historic housing stocks, rising house prices, and high rates of market-rate developments. [...] The researchers also said “the crisis is not yet half over” and that the city can expect the displacement of lower-income households to accelerate in coming years.

The interactive map is worth exploring for yourself.

All of which means that the rents in Oakland are already rapidly increasing (partially because it's within commuting distance of San Francisco). So where's next? If I'm running a small startup that needs to lengthen its runway while I figure out my product / market fit - or better yet, if I'm an artist that wants to live somewhere nurturing, affordable and creative - where can I go?

My money's on one of two places:

Sacramento.

As Thrillist noticed this summer, California's capital has a plethora of food and culture, for a much lower living cost:

We have a hard time even talking about San Francisco rent anymore. We start sweating, breaking out in hives... yeah. Especially when we think about rent in Sacramento. I mean, look at this -- $1,650 for a four-bedroom HOUSE!? That’ll get you, what... a patch of ground under the freeway in San Francisco?

Pretty much. In fact, Sacramento is 36.5% cheaper to live in than San Francisco overall (and rent is 65% less). And, yeah, it's the seat of state government, which gives enterprise startups access to a different kind of infrastructure. The only real bummer is that if you do need to get back to the Bay Area for meetings, the drive will take you two hours in good traffic.

Santa Rosa.

Situated in the middle of wine country, Santa Rosa is also adjacent to a lot of the trappings of fine living, although it's a little less hot on live music and theater. (Those needs are met by Sebastopol, just a few minutes down the road, which is also a base for O'Reilly Publishing.) Overall, it's a little more industrial than Sacramento in itself, but is set in outstandingly beautiful countryside and high-class local amenities.

But here's the big plus: as well as being super-close to Petaluma (home of TWiT), Sebastopol, Sonoma, Healdsburg and Napa, Santa Rosa is only an hour's drive from San Francisco. The Smart Train will provide effective public transport for the north bay - something it's sorely lacking right now - and further reduce the commuting pain. The first stretch, between Santa Rosa and San Rafael, opens in 2016. It'll use the same Clipper card system used by BART and Muni, effectively linking the north bay to the greater San Francisco Bay Area transit system. And expansions will link the train to the existing ferry infrastructure.

My bet is that the Santa Rosa corridor will be the next place to look. Its top-tier office space is a third of San Francisco's cost, it's surrounded by beautiful countryside and some of the country's best food, and is still within a hop, skip and a jump of Silicon Valley.

Rather than taking a retrospective look back at 2015, I think it's interesting to look ahead think about what I want my themes for the next year to be, both personally and professional. Here's mine; I'd love to see yours.

Be more social.

I want to spend more time around more people. Humans are social animals; spending more time around people has an important effect on my mood (as well as opening new horizons and opportunities).

I love people, but I spend a lot of my time behind a screen. When I'm on my deathbed, I don't think I'll look back and think, "gosh, I wish I'd spent more time on the Internet". I want to spend more time away from a screen, not thinking about work or computers, hanging out with people I care about.

And guess what: I bet it'll improve my work, too.

Be healthier.

I’ll turn 37 in the first week of 2016. As much as I hate to admit it, I have to acknowledge that I’m approaching middle age. I intend to live past 90, but that doesn’t happen by chance.

I'm also worried about what happens if I encounter a major health issue later in life. If you have to have a major operation - something many people I love have had to do - your chances of survival and recovery are much better if you're fit. It still feels a bit weird to be accepting my own mortality, but that's just stage one; stage two is embracing it.

Accept the superficial.

I was brought up to believe that appearance doesn't matter; that it's what's inside that counts. This should be true, but it isn't at all.

I know that my personal appearance affects how I feel, even aside from first impressions, but even today, I feel actively guilty for thinking about it. This is a very silly thing to be worried - particularly given what I do for a living. I could write a whole essay about all the issues at play here. It's a weird hang-up.

Be more organized.

Specifically, I'm going to start scheduling more of my off-time in the same way that I schedule meetings.

Not only will this allow me to schedule in gym time and other exercise, but keeping a tighter schedule will give me more free time for chilling out, hanging out, personal creativity, and trying new things: all vital parts of being an actual human being.

Separate creativity and work.

I've repeated this quote before: "the business of business is business". Work can be creative, which is awesome, but very few of us are lucky enough to have a job that is our creative outlet. I think if you try and shoehorn that creative need in, you run the risk of being unsatisfied both with your creativity and with your job.

I want to write more; draw more; publish short stories and write more personal pieces. And in turn, I want to be more focused in my work. What I build does not need to be a reflection of me, and in turn, who I am is not a reflection of what I build. This perhaps sounds trite and reductive, but it's important.

Be more "me".

In 2016, I want to be more political. I'm less interested in tweeting links than actually marching on the streets, phoning representatives and doing real work to support the causes and politicians I believe in. I believe in a fairer society, globally, and I'm not ashamed to want to advocate and fight for that. I want to live in a progressive society.

So politics is one thing. But I value art, and creativity, and outsider culture. I want to spend more of my time in the kinds of anarchic artistic communities I was a part of in Edinburgh. Mainstream culture is deeply conservative, and deeply boring. I think the voices pushing at the edges are usually the most interesting, and I think it's a real shame that there aren't more safe spaces for them.

I want to be more experimental in my personal expression. I want to be more supportive of the voices I value. I want to be clearer in my non-support for the status quo.

I don't think this hurts what I do (although I'd do it even if it did). In a world that's becoming increasingly algorithmic and computational, our humanity is our sustainable competitive advantage.

It's good to be a person. I want to nurture more of myself, and more of the people around me, in the year ahead.