Werd I/O

We've got some exciting new things in store for 2016 that solve real problems for both higher and corporate education. We'll discuss this in a future post on the Known blog. First, though, I wanted to take a step back and explain the technical decisions we made for Known.

What is Known?

Known is an open source web platform that allows groups and individuals to publish in a group with a variety of media. You can choose who can see the content you publish, as well as where you reach your audience: you can syndicate your content to services like Twitter, Facebook, SoundCloud, Flickr, LinkedIn and more.

It's also an open platform designed to be extended:

• Every content type is provided by a plugin, so any organization can add new kinds of content. (For example, we don't provide video out of the box - but you could.)
• Every syndication service can also be extended, so while we provide plugins for social media, a university could extend Known to allow students to submit their work to their Learning Management System. Internally in the company, we'll often create Known sites for particular projects and then syndicate our posts to Slack.
• Known supports themes.
• Plugins can also provide Single Sign On (we provide LTI and LDAP to our enterprise customers, but for example, KQED uses SSO to link Known to WordPress accounts).

Known works for a single user - my website runs on it - or five thousand. It's up to you.

Did I mention it's fully responsive, meaning it works just as well on your smartphone as it does on your laptop? Or that every page is an API endpoint?

Install anywhere, extend easily

A key goal for Known is the ability to install it virtually anywhere.

Installing self-hosted web software is, unfortunately, not as easy as installing an app on your iPhone or your laptop. However, it doesn't need to be a developer-centric process.

Shared web hosts are immensely popular, and abstract away a lot of the really technical work involved in maintaining a server. You can often select an application to install from a directory of available projects, answer a few questions, and be ready to go in a couple of minutes. At its hardest, you can upload some files via FTP. You never have to drop to a command line and run Linux commands - and indeed, often you can't.

We wanted to be compatible with these hosts (our web hosting sponsor is DreamHost), as well as power users who have deeper technical control over their servers. That implied a number of requirements:

• The software language needs to be compatible with a large number of servers
• Users need to be able to install the software without command line tools
• Knowledge of version control systems like git, or managers like GitHub, shouldn't be required
• Use of a package manager like npm or composer shouldn't be required

It turns out that the most widely-supported language on shared hosts is PHP.

PHP has received not a small amount of scorn in developer circles over the last decade, and a lot of it is fairly earned. But the truth is that modern versions - particularly 5.4 and above - have consistent interfaces, and modern language features like namespaces and closures that bring it closer in line with more cutting-edge languages. The PHP style recommendations produced by the Framework Interop Group and popularized by PHP The Right Way have done a lot to standardize PHP code.

In fact, PSR-4, which defines a template for class namespaces and a way for objects to be autoloaded on demand, turns out to be useful. Every plugin in Known uses this standard for autoloading.

The only question is PHP version: not every host supports these features. In fact, while it turns out that 98.8% of PHP hosts support version 5 or above, 34.3% of these are on version 5.3. We expect this number to shrink over time, and consider it acceptable to be supported by the remaining 65% of web hosts. The syntactic features you gain, like closures, are worth it.

To support virtual URLs, we initially required the Apache web server (which is still the leader overall on the web). However, a number of community members have created open source configurations for nginx.

The data model

I don't think it's acceptable for plugins to create and maintain their own database tables. For one thing, you may wish to prevent Known from having database modification access permissions. For another, this means that every plugin is a potential database security risk or performance drain.

Instead, from the beginning I wanted plugins to access the database via an abstracted interface, and never have to worry about the schema. At the same time, I wanted plugins to be able to store any data they needed to function, in a way that made sense in the context of that plugin.

The first versions of Known used a NoSQL database, MongoDB, as its sole data store. This worked well for development, but it quickly became apparent that shared hosting would not support this as a data layer. In interview after interview, users said they wanted to run Known on hosts like Reclaim Hosting and Nearly Free Speech. In fact, many shared hosts support MySQL - and that's it. This left us with a challenge: could we provide a schemaless database layer while providing full support for MySQL?

Kevin Marks provided the answer: a balanced schema developed by FriendFeed back in the days before NoSQL databases became commonplace. We created a highly-indexed metadata table, which is purely used for searching for objects, and then stored the complete objects in JSON in an object database. All of this is provided by a seamless database layer called the Data Concierge, that abstracted many of the functions provided by the MongoDB PHP extension.

A side effect of this abstraction is that more databases could be added easily. Today, as well as MongoDB and MySQL, Known supports SQLite and Postgres.

Distributed social networking and uncool URIs

One of the core original visions for Known was that data could be distributed. A user on site A could participate in a community on site B. Imagine creating a group for a project across two companies, and then allowing users from a second company to join and collaborate without re-registering! There are lots of real-world possibilities for distributed social networking.

To prepare for this, we decided that every object would have a URI as its definitive UUID. The idea was that you could access any resource by its UUID anywhere on the web, and as long as the request was properly signed, you'd be able to access it as if it was locally stored. In the end, I consider this a core mistake, but one that is hard to move away from.

Tim Berners-Lee famously said that "cool URIs don't change". Unfortunately, in the real world, URIs change all the time - and there's no way to require that they don't.

• Domain names expire
• People lose control over their domain names (eg a subdomain at a university)
• People choose to move sites into subdirectories

Imploring people to strongly consider their website layouts, as the W3C does, is not helpful for individuals who just want to run a site. The web is not set in stone; websites change, and URIs should be treated as volatile in any internal data model. 

As it stands, Known contains a number of protections that allow it to be moved to different domains or directory locations, so users don't notice a difference. It's not a technical decision I'm proud of - but it may yet come into its own. We already use the indie web technologies for some distributed social networking, and it's an idea that I'm convinced will transform the web.

The front end

Creating a native mobile app for a platform that can be infinitely extended is difficult. Instead, we created a fully responsive, touch-friendly interface.

Known separates model, view and controller, and any page can be viewed with a different template. For example, here's my website using a JSON template, and here's a Star Wars crawl. Any plugin or theme can override any template element, so I could write a plugin that changes out the WYSIWYG editor (we use TinyMCE), or that displays avatar images as 3D spheres (if I really wanted to). I could write a template to display Known sites using a virtual reality browser - and someone really should!

For the default template, we chose Bootstrap and jQuery. The former provides a solid, responsive UI that can be extended easily (and which removed the need to develop it from scratch). The latter provides a powerful, performant way to query elements on the page. Not only did this combination let us get up and running quickly, but plugin authors could use them to create simple, grid-based user interfaces that would be in line with the platform as a whole.

For glyphs like social media logos, we use FontAwesome. The latest version contains 605 different user interface icons, is well tested, has a good community, and a compatible open source license. All of these things made it perfect for our use - and, again, making features available to plugin authors.

Every page is HTML5, CSS3. Content is encoded using microformats, allowing software to read and extract meaning from our human interfaces. This forms the basis of important decentralized social web protocols like those used by the indie web community.

Over time, we've learned that we do need to support a mobile app. The mobile web has evolved to be decent for consumption, but there are obvious missing pieces for producing content on a mobile over the web.

For example: it's difficult to upload media. Resizing camera JPEGs in front-end Javascript on a mobile device is not a reliable process. The web audio API produces WAV files, rather than MP3s, which are uncompressed and potentially large. We could resample these on the server side using something like ffmpeg, but it's not reasonable to expect a shared host to support media encoding - and nor is it reasonable to expect users to link up to a third-party media encoder like Zencoder. Worse: we found that the web audio API actually crashed many mobile browsers!

This problem is compounded by video uploads. Video files are huge, and there's no way to compress them in a browser. Backround uploads are hugely tricky, and resuming failed uploads is also hard. That's even before they've reached the server - and when files can be as large as 1GB per minute of footage, both storage and encoding is hard.

For the mobile web to effectively compete with apps, it needs to support the content composition experiences that native apps have been using for years. If we want people to build websites, the web needs to support building, across devices. It's a frustration, and an ongoing problem.

Moving on

Our PHP-based infrastructure and need to support shared hosts means that some features are much harder to produce. The truth is that technologies like websockets (useful for performant real-time user interfaces) are hard for non-developers to self-host. New web platform features like web workers show enormous promise, but require secure connections - and even with empowering projects like Let's Encrypt, setting up secure sites is still too complicated for most people.

The good news is that some progressive enhancement is possible: companion services that provide extra capabilities to hosted software. It's also true that hosts are evolving, and our friends at DreamHost and Reclaim Hosting are thinking hard about the future of the space. 

I'm proud of the platform we've created - it's one we use every day, and I'm delighted to see people posting on their own servers all over the world. We've got big plans for the Known open source project this year, and we're looking forward to sharing them with you, in conjunction with something new that we'll tell you about soon.

It's going to be a great year.

This afternoon, Greg Wester tweeted this screen grab:

Your eyes are not deceiving you. That's a 420 square foot studio apartment - not even a one bedroom - for $3,050 a month. That's a base cost: your bills, and most likely parking, are extra.

There are only two kinds of people who can afford this:

• Workers on a substantial salary (realistically $100K+, which would still leave you with a bi-weekly check of a little under $2500 after taxes, meaning rent would be well over 50% of your monthly cost)
• The independently wealthy

Let's leave aside the obvious social inclusion issues at play here, and the effect this has on diversity in the city. Let's ignore that this is killing the artistic temperament of the city and turning it into a primarily financial center like any other. Let's pass over the inevitable effect this will have on the city when these high-value residents start to ebb away. Let's pretend not to see the rising homelessness problem. Not because those are unimportant issues - they're vital to the future of San Francisco - but because it's harmful to the ecosystem that helped create this situation to begin with.

If only rich people can afford to live in San Francisco, it is impossible to really innovate. All the creative energy is being driven out. There's no way for ordinary people - people who haven't made it yet - to experiment. Everyone is either on salary or has raised money from institutional investors with a proven business model.

So the gentrification cycle turns:

Deindustrialization creates low-rent vacancies in industrial districts; artists are drawn to these districts by the depressed rents and spacious "lofts"; the district becomes a hub of avant garde creativity, generating media attention and foot traffic, both of which create a "buzz" around the neighborhood; shops and restaurants are drawn to the area to cater to the increased foot traffic and capitalize on the "buzz;" the introduction of these shops and restaurants in turn induces more foot traffic, more media attention, and more "buzz;" eventually national chain stores see the area as ripe for investment and begin to move in; finally, of course, each of these trends causes rents to escalate until, with the arrival of deep-pocketed chain stores, the very artists who made the district trendy are priced out.  The district ends up as nothing more than a high-end outdoor shopping mall with little street "cred," and the artists relocate to a new low-rent industrial area, triggering the process all over again.

We've seen this process start to rapidly transform Oakland:

Oakland neighborhoods that are experiencing "advanced gentrification," according to the study, include Lower Bottoms, Old Oakland, and Northgate/Koreatown. The researchers define "advanced gentrification" as areas that have experienced significant demographic changes and high levels of real estate investment. Those areas are also very vulnerable to gentrification due to their locations near transit, historic housing stocks, rising house prices, and high rates of market-rate developments. [...] The researchers also said “the crisis is not yet half over” and that the city can expect the displacement of lower-income households to accelerate in coming years.

The interactive map is worth exploring for yourself.

All of which means that the rents in Oakland are already rapidly increasing (partially because it's within commuting distance of San Francisco). So where's next? If I'm running a small startup that needs to lengthen its runway while I figure out my product / market fit - or better yet, if I'm an artist that wants to live somewhere nurturing, affordable and creative - where can I go?

My money's on one of two places:

Sacramento.

As Thrillist noticed this summer, California's capital has a plethora of food and culture, for a much lower living cost:

We have a hard time even talking about San Francisco rent anymore. We start sweating, breaking out in hives... yeah. Especially when we think about rent in Sacramento. I mean, look at this -- $1,650 for a four-bedroom HOUSE!? That’ll get you, what... a patch of ground under the freeway in San Francisco?

Pretty much. In fact, Sacramento is 36.5% cheaper to live in than San Francisco overall (and rent is 65% less). And, yeah, it's the seat of state government, which gives enterprise startups access to a different kind of infrastructure. The only real bummer is that if you do need to get back to the Bay Area for meetings, the drive will take you two hours in good traffic.

Santa Rosa.

Situated in the middle of wine country, Santa Rosa is also adjacent to a lot of the trappings of fine living, although it's a little less hot on live music and theater. (Those needs are met by Sebastopol, just a few minutes down the road, which is also a base for O'Reilly Publishing.) Overall, it's a little more industrial than Sacramento in itself, but is set in outstandingly beautiful countryside and high-class local amenities.

But here's the big plus: as well as being super-close to Petaluma (home of TWiT), Sebastopol, Sonoma, Healdsburg and Napa, Santa Rosa is only an hour's drive from San Francisco. The Smart Train will provide effective public transport for the north bay - something it's sorely lacking right now - and further reduce the commuting pain. The first stretch, between Santa Rosa and San Rafael, opens in 2016. It'll use the same Clipper card system used by BART and Muni, effectively linking the north bay to the greater San Francisco Bay Area transit system. And expansions will link the train to the existing ferry infrastructure.

My bet is that the Santa Rosa corridor will be the next place to look. Its top-tier office space is a third of San Francisco's cost, it's surrounded by beautiful countryside and some of the country's best food, and is still within a hop, skip and a jump of Silicon Valley.

Known has become the easiest way to create an online community to support your class or group. We've built an easy-to-use platform that lets people publish in a group with a variety of media, from blog posts and photographs to files and points on a map. Each post can be private or public; every Known site as a whole can be private or public. And it all works on any device, from the biggest, strongest desktop to the most entry-level smartphone, as long as it comes with a web browser.

Institutions like Harvard and MIT use it to run classes; so do groups teaching web skills in rural India, activists promoting racial justice, writers who need to control their identities, and open source hackers.

Here's how we got here, and here's where we're going.

Finding a fit in higher education

We arrived at Matter knowing we wanted to give people more ownership over their conversations and content online. As well as investing in our team and creating a structured environment for us to grow our company, they gave us a grounding in design thinking which helped us change the way we think about technology businesses.

It was through this process, and hundreds of hours of conversations with teachers and students, that we discovered a deep need in education for social platforms. 98% of higher educational institutions use something called a Learning Management System - platforms like Blackboard and Moodle - but very few report that they are satisfied with the experience. These platforms focus on administration, rather than learning. While they are often used for classroom teaching, they fall comically short of the kinds of social experiences students are used to.

Enter Known. Our platform runs as a stand-alone community site, but it can also integrate with a school's LMS to add those much-needed social features. We offer single sign on to campuses, and unlike many social platforms, let you publish any kind of file you need to. All our plans come with unlimited storage and bandwidth, so you don't need to worry about capacity. We sell SaaS subscriptions, and enterprise licenses for organizations that want to run Known on their own infrastructure.

We also understand that conversations don't just happen in tiny sites on the web. Known sites can push their content across social networks: audio, for example, can be immediately copied to a SoundCloud account. Using brid.gy, we can pull replies and likes from those social networks back to the community, so everything is always stored in one place.

Social infrastructure for campuses

The possibilities are endless. Any campus can run as many Known communities as they need to. We also know that discovering all the content being created on a campus is key, so we've started to provide social hubs and search engines for all of it. On-campus users can search for content that only they can see; visitors to a campus can search for and discover content that has been made public. The result is an easy-to-use gateway to everything happening at a campus. It's never been done before.

We know that in education, one size rarely fits all. So we offer design sprints, where we'll arrive on campus and run design thinking sessions with students, faculty and staff. These allow us to tailor the product to meet the needs of a particular institution, so it complements their activities, their design, and their culture. (These sprints turn out to be useful whether you end up using Known or not.)

Because that's the other thing about Known: it's open source and extremely customizable.

An open source core

In VentureBeat, Lightspeed's John Vrionis writes:

The OSS companies that will be pillars of IT in the future are the companies that leverage a successful OSS project for sales, marketing, and engineering prioritization but have a product and business strategy that includes some proprietary enhancements. They’ve figured out that customers are more than happy to pay for an enterprise-grade version of the complete product, which may have security, management, or integration enhancements and come with support. And they also understand that keeping this type of functionality proprietary won’t alienate the community supporting the project the way something such as a performance enhancement would.

This is our strategy. Our core platform is available on GitHub: you can get it right now. We offer a fully-managed service, with unlimited storage and bandwidth, so you don't need to worry about server maintenance or capacity. But we also offer premium features like LTI integrations, file uploads, and searchable user directories.

We love our open source community. Thousands of people use Known to publish on their own site as an indieweb blog, and the activity helps us build a better platform for everyone. Every single page on every Known site has a little heart icon. Click it, and you're prompted to send us feedback. We read every single message personally, and it allows people who aren't developers or designers to contribute to the community and help us develop the product.

John goes on to say:

OSS businesses turn the customer discovery process completely upside down. Open source software is put into the wild, and the company immediately receives signals from those who are interested. Entrepreneurs get the benefit of real data and usage to help them decide where to focus engineering and sales-and-marketing resources. This is tremendously helpful and important. Data, not guessing, drives prioritization of the limited resources at a company’s disposal.

The combination of an open source development model and a design thinking product process means that we can rapidly prototype new ideas, and get strong signals from real people about the desirability of our platform.

Beyond education

It's obvious that a flexible community platform that runs on any device has applications beyond education. With LDAP / Active Directory integration, you can run it alongside your intranet to support a project or a company. Because you can make a community private, we've even seen families use it to share photos of their children that they wouldn't feel comfortable publishing on Facebook.

Mozilla's CEO Chris Beard said today that he thought of revenue as "a means to do better for the world". We agree: it is important to be a growing, valuable company, but in service to being able to provide a platform that can support any class and give anyone in the world a voice in a space they control. The total market for Known in education is measured in billions of dollars, but our potential goes beyond that.

We're living in a world where everyone can be connected, but only a handful of companies control those conversations. Censorship and surveillance are growing threats. By creating an open, easy-to-use platform that works on every device, we can help everyone own their own conversations. Not only can top-tier universities and companies benefit, but we can help disadvantaged communities, too. From non-profits sharing resources in developing nations to vulnerable groups who need to protect their identities right here in America, we believe we can make a difference.

Our role as technologists is to build a better future where everyone is represented. That's the promise of the web, and it's something core to our mission and beliefs. We're building what I call respectful software, and by showing it can be successful, we will encourage other vendors to follow.

Today, it's the best way to build an online community. But Known has an even brighter future ahead of it. We're excited to bring it to you.

Get involved

Check out our website, and follow us at @withknown on Twitter.

If you're a developer, you can find our core platform on GitHub, and you're invited to join the developer mailing list.

And you can always email me at ben@withknown.com. I'd love to talk to you.