Werd I/O

TechCrunch reports that IBM bought Aspera yesterday:

Aspera’s software is built on “fasp,” its patented file transfer technology. Fasp is designed to leverage a company’s wide area network (WAN) and commodity hardware to achieve speeds that are faster than FTP and HTTP over a secure network. A WAN is essentially a company’s network across a large geographic region. Aspera’s technology optimizes the WAN through its software that allows for granularity in the way the technology is used. Through the process, Aspera optimizes the bandwidth, latency, bottlenecks and a host of other factors.

Essentially, Aspera is an optimized, proprietary protocol built on UDP. The sender and receiver needs to have Aspera software (or software that licenses Aspera's technology) installed for the transfer to take place.

IBM's thesis that we need faster file transfers is obviously correct. Uploading large data or media files is a pain, and it's not a problem that's going away. While downstream bandwidth is getting better all the time, upstream bandwidth often suffers in comparison - and between big data, more sophisticated applications and our insatiable appetite for video, the size of the files we're moving across the Internet is going up.

[Disclosure: right now I'm CTO at latakoo, a startup that helps enterprises (including TV news networks) move large amounts of media data around. We use a toolbox of compression techniques, as well as a global upload network, to make managing media files much faster. Journalists use us to, for example, upload video from Air Force One while the plane is taxiing down the runway.]

Don't think there's a problem? 5 minutes of recorded video on an iPhone 5 can be 1gb or more in size. Try uploading that from anything other than a super-fast broadband connection.

The trouble is that a protocol like Aspera's naturally leads to lock-in, and can only solve the problem for certain kinds of software. Unless fasp is open-sourced now that it's owned by IBM, every piece of software you own will need to have a license for their patents in order to take advantage of it. Not only does that lead to more expensive software, but it also limits the innovation that can happen around the protocol. If your business relies on fast file transfers, that's a lose-lose situation for you: you're locked into an expensive ecosystem, and there's little chance of a disruptive incomer to tweak and play with the protocol. Because of that, folks will just invent new protocols. When an improvement is made, it's likely you'll have to buy into a whole new ecosystem, as opposed to just upgrade to the next version. You'll hop from lock-in to lock-in.

Knowing that we need to speed up uploads, and use all available bandwidth, we need to think about other, more open approaches. UDP makes a lot of sense as a bedrock, but an open protocol designed for resilient file transfer is needed - and then, with a first version prototyped, we need to seed libraries all over the place, in every available language. We need to design it for backwards-compatibility. And we need to make sure it remains unencumbered with patents (possibly, in today's climate, by defensively registering a patent and then widely granting a license).

Our approach with Latakoo has always been to use standard protocols and optimize the files that are being transferred over the top. That works really well, and ensures wide compatibility with all kinds of networks, from corporate infrastructure through satellite modems connections. Even better would be to be able to rely on a well-supported open protocol that optimizes the upload stream in ways that TCP does not (and then, continue to optimize content with this as a starting point). Bittorrent exists, of course, and is wonderful - but is often blocked because of unfortunate connotations and liabilities relating to media piracy.

Developing such a protocol is in the interests of Netflix, Google, and, yes, Latakoo. Faster file transfers are empowering for every user, and save large service providers money. They're easier to integrate with, and allow for an ecosystem where the customer is in control. Everybody wins.

Back in the old days, you'd get a newspaper delivered, and you'd perhaps read it around the breakfast table or at your desk to find out what was going on in the world. The same paper, day in, day out. You'd read the columns, know which comics they ran, maybe do the crossword.

Some people still do this. That's nice. There's something great about reading a paper over a cup of coffee in the morning.

That doesn't apply to the web. Here's the Independent's paywall message:

Thank you for reading and relying on independent.co.uk for your news and information. You have now viewed your 30-day allowance of 5 FREE pages. Want to read more?

I'm sorry, The Independent, but I don't rely on you for anything. I don't read a linear paper on the web; I follow links curated by people I trust. The New York Times has no such illusions, meanwhile, but tells me, in all caps:

YOU’VE REACHED THE LIMIT OF 10 FREE ARTICLES A MONTH

Thanks, The New York Times. There's no need to shout.

The Independent's US price is $3.99 a month. The New York Times is a little more complicated, and expensive: $3.75 a week for the web and my phone, $5 a week for the web and my iPad, $8.75 a week for the web and my phone and my iPad, or all of the above plus a pile of dead trees every morning for the inexplicably lower price of $3.40 a week.

The New York Times is my favorite newspaper on earth, but imagine if I did this for every source I read on the web! I'd be broke, instantly. This is a model that scales well for the dead-tree economy, but doesn't work at all on the Internet, where you could easily read 10 sources in just a morning.

Hence advertising: the Independent knew I was unlikely to buy, so actually displayed a full video ad next to the advertisement asking me to subscribe. It clearly wasn't contextual, because it was for a Porsche - so it's a shot in the dark, basically. Awareness advertising with no real metrics (I assume) to back it up.

Another model must be found.

I don't think it's micropayments. Imagine if you had to pay for every single thing you read on the web, which is the future that micropayments promise. Just as if you paid for a subscription for every newspaper site on the web, if you're anything like me, you'd be broke pretty quickly. Or, alternatively, the payments would be of the kind that we've seen on subscription music services. That's a road that leads directly to Buzzfeed, where articles must be massively popular to turn a profit - and hence are impossibly populist.

In the old days, classified ads played more of a role. It's true that Craigslist's success disrupted $5 billion from the newspaper industry, but it's also true that this does not explain the hard times newspaper owners are experiencing. The context has changed, and attempts to drive traditional subscriptions show that the industry still hasn't full adapted to this. It's not that we're reading on multiple devices now, in different locations - it's that we're reading differently. The newspaper front page hasn't been our first stop in a long time.

Obviously, none of this is, well, news - countless articles have been written about this over the last decade. Yet, a solution has yet to be found. Which is a shame: journalism, newspapers and the communities they represent have become an important, and I'd say integral, part of the world in which we all live.

Back in 2009, I wrote:

Right now, we have to register with each application we want to use. What if we required each application we used to register with us, in digital identities under our own control?

What if, using these identities, anyone could connect to anyone else, and anyone could store their data anywhere as long as the storage provider followed the same broad standards?

The web itself would become a social networking tool.

By establishing a general standard for social application interactions, the services and technologies used to make connections become less relevant; the Internet is people, one big social network, and users no longer have to worry about how they connect. We can all get on with communicating and collaborating rather than worrying about where we connect.

The full piece was based on a talk I gave at Harvard University's Hauser Center for Nonprofit Organizations. It declares a number of problems to solve:

• User control: users should have ultimate control over their data.
• Ownership: granularity of ownership, and the rights implied by ownership, is complex in certain contexts. Sometimes a single-user model of ownership is appropriate - but sometimes not.
• Privacy & Transparency: the ability to control who has access to your data footprint across the web, but also a clear knowledge of what happens to that data.
• Platform: software that actually embodies these properties.

In the interim, there have been many articles about the continuing silo-ization of the web - notably The Web We Lost by Anil Dash. In other words, the problem has become worse, not better. Generally speaking, users have less control, less ownership, less privacy and fewer platforms to choose from in 2013 than they did in 2009.

A glimmer of hope has been the indieweb, which I've written about at length. This is a movement that champions ownership, but through it, principles like user control, privacy, transparency and a healthy ecosystem of platforms are also promoted. Idno, the open source platform that powers this site, adheres to many indieweb principles.

There's more work to be done. I believe that contextual display advertising is the single biggest obstacle to a web that is under the control of users. In our advertising economy, users are tracked throughout the web in order to determine which ads will be performant for them. Mozilla Lightbeam is an extraordinary project that highlights the pervasiveness of the problem. Wherever we leave a data footprint, we are tracked.

The irony is that contextual advertising isn't even very effective! Fraud is rife in online advertising, and the price of online ads has dropped for eight straight quarters. As a result, publishers need to drive higher and higher visitor numbers, leading to less subtle growth strategies, often bordering on the unethical. Platforms seek vastly increased engagement, leading to an inability to remove your content, what amounts to spamming you to bring you back to the app, and a reduction in integration hooks that might make the software more useful within the context of a user's entire suite of applications.

On the content side, meanwhile, viral sites like Upworthy and Buzzfeed are king, which is great if you're writing about the top 15 things you might not know about Miley Cyrus - but death if you're a niche publisher, community or information source trying to make ends meet.

What if we rethink advertising in the same way that we're rethinking personal sites with the indieweb? "Niche" - in other words, highly specialized - communities are in many ways the lifeblood of the web. They're one of the things that makes it special; the fact that there can be a place to meet for any interest group. Finding platforms that will adequately financially support these groups, as well as by giving them responsible software that gives them control, privacy, transparency and ownership, will be hugely empowering.

Building the open web we want isn't just about software. It's about the mechanisms involved that will make it sustainable for people to create the right kinds of businesses that use it as a platform.

#indieweb #communities

Edward Snowden released this letter to the world in PDF format. The following is a searchable text version of same. Any errors or inconsistencies are unintentional and my own. I am unconnected to Edward Snowden and he has not authorized this reshare or transcription.

To whom it may concern,

I have been invited to write to you regarding your investigation of mass surveillance. I am Edward Joseph Snowden, formerly employed through contracts or direct hire as a technical expert for the United States National Security Agency, Central Intelligence Agency, and Defense Intelligence Agency.

In the course of my service to these organizations, I believe I witnessed systemic violations of law by my government that created a moral duty to act. As a result of reporting these concerns, I have faced a severe and sustained campaign of persecution that forced me from my family and home. I am currently living in exile under a grant of temporary asylum in the Russian Federation in accordance with international law.

I am heartened by the response to my act of political expression, in both the United States and beyond. Citizens around the world as well as high officials - including in the United States - have judged the revelation of an unaccountable system of pervasive surveillance to be a public service. These spying revelations have resulted in the proposal of many new laws and policies to address formerly concealed abuses of the public trust. The benefits to society of this growing knowledge are becoming increasingly clear at the same time claimed risks are being shown to have been mitigated.

Though the outcome of my efforts has been demonstrably positive, my government continues to treat dissent as defection, and seeks to criminalize political speech with felony charges that provide no defense. However, speaking the truth is not a crime. I am confident that with the support of the international community, the government of the United States will abandon this harmful behavior. I hope that when the difficulties of this humanitarian situation have been resolved, I will be able to cooperate in the responsible finding of fact regarding reports in the media, particularly in regard to the truth and authenticity of documents, as appropriate and in accordance with the law.

I look forward to speaking with you in your country when the situation is resolved, and thank you for your efforts in upholding the international laws that protect us all.

With my best regards,

Edward Snowden
31 October 2013

A while back, my out-of-warranty Macbook Pro's wireless antenna died. I had the card replaced; no dice. And unfortunately the antenna is in the screen, so to get working wireless back I would have had to get that replaced at a very high cost.

I went another way.

If you walk down the Networking aisle at Best Buy or Staples, which is what happens when you need a new wireless adapter in a hurry, you'll see that all the USB wireless dongles say they're for Windows only. Not a single one ships with a Mac driver, because I guess the Mac hardware is considered to be infallible or something. Maybe the cost-benefit analyses didn't show it to be worthwhile. Who knows.

Luckily, most USB wireless dongles are repackaged chipsets from other manufacturers. In particular, the Linksys AE3000 and a few other wireless-N models by manufacturers like Belkin are based on the Ralink (now Mediatek) RT3573 chip. I bought the Linksys AE3000 because of the build quality and speed capabilities.

Years ago, Ralink released some official drivers for Hackintoshes, which they've kept up-to-date. So after buying my Windows-only Linksys dongle, all I had to do was go grab the appropriate driver from their download page.

Or so I thought. You see, it turns out that the AE3000 didn't exist when the driver was written, so the installer doesn't know anything about it. You plug in your dongle, and nothing happens. What you actually have to do is install the driver and, before the final reboot after installation, go find the Info.plist file in /System/Library/Extensions/RT2870USBWirelessDriver.kext/ and add some information about the manufacturer. (After some adventures with text editors, I found that it was best to do this using sudo nano in a terminal window.)

And this kind of ridiculousness is why open source operating systems are a good idea. Nonetheless, despite the convoluted technical steps, it works: I have a working wireless-N connection via my Windows-only wireless dongle (even after an upgrade to Mavericks). I hope this helps someone else - and that manufacturers start properly supporting Mac OS X.

The BART strike is over: at the time of writing, they're expecting Bay Area Rapid Transit trains to start rolling at 4am, with a full service up and running by sometime in the afternoon. That's great news: the roads have been gridlocked here, and it's felt like the Bay Area has been brought to a standstill.

Nonetheless, I fully supported it.

The base salary for a train operator is $56,000 - a lot in most areas, but when a one-bedroom apartment runs for almost $2,000 a month, it's a salary that doesn't stretch very far. Particularly not if you have a family. And given that BART was running on a budget surplus, and worker wages were frozen for five years, it seems very reasonable to ask for more.

A second issue was safety: workers were asking for better lighting, and better protection against attacks. This request is all the more poignant given that two workers were killed on Friday.

I'm pleased that the union and BART management have reached a deal, not least because it was inconvenient to me: BART is by far the best way to travel around the area. What I'm less pleased about is the amount of anti-union propaganda I've seen from all over Silicon Valley. From tasteless jokes to threatening to replace them all with robots, it's not been pretty.

The purpose of unions is to allow workers to collectively organize and deal in a way that they could not as individuals. A company must negotiate for its best interests, by attempting to get the best value out of workers. It makes sense that the workers should have their own ability to negotiate with similar weight. Without this ability, wages, benefits and working conditions will tend to favor the companies rather than individuals. Here in the US, the labor movement was responsible for establishing the 40-hour workweek and the concept of having the weekend off, which were only ratified in 1940.

Unions have also been responsible for establishing the minimum wage, the concept of sick days, holiday pay, maternal leave, child labor laws and laws eradicating sweatshops in the United States. None of these are at all bad things, and while unions are not always a positive thing - just as company management is not always a positive thing - I'd argue that they're an important part of the fabric of working life. I have been proud to be a member of unions in the past, and if there was an appropriate tech industry union, I'd be proud to be a member of one now.

As Politico points out, a Harvard / University of Washington study showed that between a fifth and a third of the dramatic increase in income inequality in the united states (40%!) is related to the decline in union membership. While it's not the single cause, it's certainly hard to ignore, and points to a larger issue related to the evolution of workers' rights (and the perception of workers) in American society.

In January, Time noted that:

First, the fact is that when unions are stronger the economy as a whole does better. Unions restore demand to an economy by raising wages for their members and putting more purchasing power to work, enabling more hiring. [...] Second, unions lift wages for non-union members too by creating a higher prevailing wage. Even if you aren’t a member your pay is influenced by the strength or weakness of organized labor. The presence of unions sets off a wage race to the top. Their absence sets off a race to the bottom.

A victory for unionized workers, then, is a victory for all of us. Why on earth should a progressive industry like technology be against better conditions and pay for workers? I agree with Michel Hiltzik in the LA Times:

Blaming the workers for the impasse is a peculiarly one-sided interpretation of what's happening. Sure, you could say that 2,400 non-automated, human employees stand in the way of Silicon Valley's determination to "build something." But it's equally true to say that BART's nine board members and its general manager are the real obstacles to a settlement. Maybe Silicon Valley should figure out a way to automate them.

We're supposed to be making things better. As an industry, we may need to rethink what that means.

#bartstrike #labor #unions

One often-seen trope in tech industry commentary is: "the geeks will inherit the earth".

It's a nice idea, which appeals to a lot of people in this ecosystem; there's even a lovely symmetry to the idea that the people who were probably bullied and ostracized in school, at least to some extent, are the same people who go on to change the world in meaningful ways. And in a lot of ways, it's true.

It's not because geeks are in any way better, or more intelligent, people. Instead, my theory is that it relates to being an outsider. People who don't change themselves to fit in are, almost by definition, more likely to think independently. If an effective leader is one who creates stand-out strategies and is able to creatively and intelligently react to circumstances, it makes sense that independent thinkers would fit the role more readily. The popular kids at school are much more suited to be followers - they've essentially taught themselves how to follow fashions rather than create them.

MBAs are not traditionally good at startups for similar reasons. They've been taught cookie cutter business methods, which make much more sense as management tactics in larger businesses than in the do-what-you-have-to context of getting something off the ground. Here in San Francisco, arguably tech startup central, besuited MBAs are often thought of as not bringing much to the table.

But geeks have their own popular kids now. Startup culture has created its own norms; brogrammers swarm San Francisco and cities like it, following the fashions dictated by outlets like TechCrunch and PandoDaily. Not a single one is likely to change the marketplace, let alone the world - and with them comes a pervasive culture of entitlement and even bigotry that isn't a million miles away from the cool kids.

Outliers are always going to be the people who bring about real change: people who can't easily be described, and whose actions can't easily be pattern-matched to an archetype. Often, these are people who don't take direction well. They might come across as weird, or antisocial. But their ideas are like nothing you've ever heard, and given the tools, they will create things you've never thought of.

Someone once said to me, in reference to someone who they thought was weird, "one way of looking at it is that they don't think mainstream culture is good enough for them." Damn straight. Being mainstream shouldn't be good enough for any of us. There's nothing to be gained by trying to be like everybody else, or by fitting yourself into a pre-defined pigeonhole.

Me? I like weird people, and I like working with them. I wish I was weirder myself: it's a sign of creativity, independence, and intelligence. San Francisco has a name for people who follow; they're called "normals" - or, sometimes, "consumers". It's not a label to aspire to.

In fact, none of us need to be normals or consumers. Once upon a time, we were needfully forced into demographic categories, so that products and media could be created that would broadly appeal to us. The Internet has created a world where anyone can connect to anyone else, whether it's to talk, to inform, or to sell. Fashions of all kinds are meaningless in a world where products can be viably created for an audience of one. They're an artifact of the age of broadcasting; one that's long since gone. We're in a post-demographic age, and if you're still trying to follow the crowd, you're a decade behind.

#startups #outliers

When I left Elgg in 2009, I immediately started working on Outmap, a social geodata platform that you could use with any web browser. It never saw the light of day.

Outmap allowed you to collaboratively create map layers, either privately in a group, or in public. Here's two use cases that illustrate what you could do with it:

• Crowdsoucing useful free wifi hotspots Back in 2009, finding free wifi that worked well was a mission. Suddenly, you weren't alone: you could set up an Outmap space, and tell it to watch a hashtag, for example #freewifi. Users could then tweet with an address with that hashtag, or add geo-information to the tweet itself. Outmap would also watch Flickr for that tag, and check for location information either in the Flickr metadata, or in the EXIF data in the photo itself. The data then could be mapped, or simply displayed in a list based on your current location.
• Gathering (or crowdsourcing) scientific data Users could add fields with types. For example, if you were doing wildlife counts, you could take your GPS-equipped smartphone into the field, and as long as it had a web browser that supported the Javascript Geolocation API, that would be all you'd need to record a result. Your Outmap space would tie your numeric recordings of wildlife data to the location where you recorded them. And because it was all social, and tied to individual user accounts, you could examine (or even remove) recordings made by particular individuals for full accountability.

It was based on data tied to individual location points, but further developments would have allowed you to group points into areas, in order to better support some scientific applications. And again, it was all web-based used existing web standards, all social, and (like Elgg) had per-item access permissions.

Outmap couldn't be released for reasons I won't go into here, and were unrelated to the mechanisms of the platform (but were related to the fallout from my decision to leave Elgg). Let it suffice to say, it was out of my hands. I abandoned work on it in 2010, and moved on to work on latakoo.

Geolocation has evolved since 2009, and I think we all now understand that the web is something that we can access from anywhere, and that pages can know about your geographic context. Some other use cases were covered by Google Maps and (particularly) Findery, which is building up a world of memories, found objects and resources on a shared map. Geoloqi, whose founders I really like and respect, supercharges the kinds of use cases Outmap supported (as well as many others). Meanwhile, there are (and were) professional GIS platforms that are extremely powerful and used for many scientific and industrial applications.

I still think there's a place for this kind of functionality in an enterprise social platform. Luckily, I'm now in a position to work on that. Watch this space.

#outmap #idno #elggmistakes #geolocation