Beyond Activities: Cross-Device Sessions

Update: I’ve scheduled the BoF session for thursday, 14:00 in room 1.404/1. I’ll pop into the systemd bof on wednesday, too.

Update 2: It turns out systemd is solving an entirely different problem. :) there’s no overlap between it and sessions-as-in-restoring-windows.

I’ve been trying to write this blog post for months, and always hitting writers’ block or other distractions. So, screw it – I’m just going to start writing and see what comes out. :)

What this is about, is sessions, and XSMP, and wayland. Activities use XSMP, the X Session Management Protocol, to save and restore groups of windows. Before that, it was only used for the login session. It’s actually a better protocol than people give it credit for, and it served well for a few decades – but times have changed. If we want to move forward and do awesome things like sharing sessions between devices, we need something new. Even Activities as they are now push the limits of XSMP – there are a few ugly little hacks hiding in there that I’d prefer not to have. :)

Now, the key here is, if we’re going to replace an ancient-but-reliable technology with something new, we want the new one to be not just better but a lot better. Something worth switching to. Small issues like XSMP’s lack of autosave will be easy to correct; the big one is that it’s process-based. Session keys are handed out per-process, and when things get restored, they’re done by calling that particular binary with the session key as an argument. This is the source of two major pains; the first is that when a process has multiple windows (like konsole), you can’t properly put those windows in different sessions – so Activities have to do ugly things to hide this. The second, and larger problem, is that it’s ridiculously non-portable and opaque. What if the program’s binary gets moved elsewhere? what if the user switches to a competing app? And of course, their phone is not going to have the same programs installed as their laptop! Even if they have a meego phone and use Calligra on all their devices, the binary for the phone will be named differently. :P

So, what do we do? We make it resource-based! :) Store data in a standard place and standard format, so that the session manager, instead of seeing “this session had /usr/bin/firefox, /usr/bin/konsole and /usr/bin/okular” sees “this session has web pages X Y and Z, which this device last used firefox to open, with X and Y in the same window; two terminals, which this device last used konsole for; and foo.pdf on page 3, which this device last used okular for”. Then if okular isn’t available it can ask the system for a program that handles pdf, and when the user sends the session to their phone, it doesn’t matter that there’s a different pdf reader, or even that the phone browser doesn’t support tabs. Not only that, but if there were 20 urls open instead of three, the phone could go “whoa, that’s a bit much; I’ll just open the first three and display links to the rest somewhere.” And I could go “hmm, when the heck did I create an activity called ‘shambles’? wtf is in it?” and get an answer without actually having to load it. ;)

Another side of this is, well, the small stuff does add up. I just heard that OSX gained some decent session support; they’ve taken lessons from the iPhone and applied them to the desktop, making a system where it doesn’t matter if an app crashes – the OS may even kill it whenever it pleases – the session support is so good it’ll come back just as it was. I’d love to see that sort of solid session support in linux; KDE apps with XSMP are pretty good right now, but they could be better (and an autosaving protocol would go a long way towards fixing that).

Now, replacing XSMP isn’t an entirely new idea. Gnome people have been sick of XSMP for quite a while, although I don’t think anyone stepped forward with an alternative (and for their purposes, it does work, anyways). When I talked to the Nepomuk guys many months ago, about cross-device sessions, they had already thought of it – but decided the political side of it was too hard. They might still be right – we’ll see. But it’s worth a try. :)

So, what is the hard part, you ask? It’s the reason I stuck with XSMP for activities: a new protocol means persuading apps to support said protocol. XSMP’s support, while patchy in places, is at least decently widespread and un-controversial. A new protocol… well, I’m fairly sure KDE would embrace it, but I want it to be a part of gtk apps too, and pure qt apps, and meego apps, and even the weird proprietary fringe ones (like maple) someday. That is a challenge. :)

So how do we meet this challenge? First, we explain the benefits it brings – I hope I’ve done that well enough above. Second, we offer app developers a well-designed, easy-to-use API that’s had some feedback from actual app developers; just the other day I saw someone on IRC complaining about XSMP being confusing. :) Third, once we’ve got enough support to know this is worth trying, we show them a solid implementation that works (without any fatal bugs). Seeing is believing, right? Nobody wants to spend precious time writing code for a system before it exists, so we get the minimal feature set done, port a couple of apps, and show it off. Fourth – the secret weapon. ;) Wayland is gaining popularity, and when apps start porting to it, they may as well port to this too, right? Wayland doesn’t have any session protocol of its own, so this could be it. :)

So, who’s with me? :) The desktop summit is only a few days away, and surprisingly I will make it (yay!) so I’d love to have a conversation about this there. It’s the perfect place, after all, for something that aims to be cross-device, cross-desktop and all.

TL;DR: a resource-based session protocol would let us do really awesome things, so it’s worth the effort to replace XSMP

Now, I’m going to go into technical details below the cut (I have enough material to write a paper on this but for that damn writer’s block…)

ContextJournal: an activity-aware app

Well, I haven’t blogged in quite a while, have I? I almost blogged about fosdem, but I was too busy having fun there. ;)

I, well, I’m actually on hiatus at the moment. That was (and is) an incredibly hard decision, but… well, it wasn’t fun any more, it felt more like a job. And when someone’s asshat regressions cause you physical pain, it’s time to take a step back. I’m sure Ivan and Marco will fix up whatever bugs are found, and add new awesome activities stuff too. :) I miss lots of KDE people (hugs!!!) but there’s some stuff I need to do for myself right now.

Anyways, that’s not what I’m here to talk about. There’s a little project I did in january that I never got around to blogging about. It’s part personal tool, part experimental sandbox. I suck at names, so I called it ContextJournal. :)

What is this? it’s a neat little app that automates away the tedium of my personal journal system. Running KDE trunk, and having a strange knack for hitting freaky PIM bugs, I end up with my most important things stored in plain text. It’s grown into quite a system, one file per day with ascii todolists and stuff – add to that the need to do timetracking on one project, and, well, a bit of automation does wonders. :)

Contextjournal allows me to keep the plaintext system – if I trash my kde or X I can still go back to using vim – but it adds improved organization, and context awareness (this is where the cool experimental bit comes in). It timestamps every line, and keeps a separate journal for each activity – naturally it switches when I switch activities. :) Once the journal part was stable I added the todolist on the right, which is controlled by irc-like commands. The commands are logged too, so finding out when I finished something is just a matter of grepping. :) Oh, and it starts on all activities by default, since it’s designed to be a systemwide thing.

Obviously the program isn’t much use as a journal to anybody but me. A normal person would probably want it to link into akonadi and/or nepomuk, and share the data with other apps, so that the todolist shows up in korganizer’s calendar n’stuff. :) However, it is useful as example code: it shows how to make an app activity-aware. It’s actually a pure-qt app at the moment too; I wanted to show that you don’t need any kde dependencies to write activity-aware code (*cough*firefox*cough*). I also wanted to have a testbed for some strange XSMP problems that only qt – not kde – has. I’m still not sure whether the bugs lie in Qt or in kde’s xsmp handling or in kwin, but my bet is on QApplication. :/

I have plans to add other shiny things, like an overview screen (or just “what now?” suggestions from the todolists) and grouped todos, but something distracted me and I haven’t gotten around to it yet. Maybe when I don’t have any essays to write. :P

Anyways, the ContextJournal source is here; if you’re thinking about adding activity support to something, check it out. And keep in mind that by now, the API for working with activities is probably in kdelibs; it’s a lot more convenient than having to make all those dbus calls yourself. :)

Activities in Action

It’s screencast time again! :)

For this one, I focused on how I, personally, use activities. I got the impression that showing real-world usage might make it a bit clearer what I’m trying to do here. :)

You can probably hear me getting a bit tired at the end… I had to reboot twice and fix my damn volume control before finally getting that one, so yes, I *am* tired. :P

oh, and here’s the activity manager plasmoid I mentioned. :)

Activity-oriented vs Application-oriented workspaces

I don’t normally comment on other people’s software… but… a little while ago, opensuse was wondering whether KDE apps would work with these Gnome Shell requirements. Last time I checked, their “Activities” sounded like virtual desktops, but now they’ve returned to an application-based shell. This in sharp contrast to Plasma Desktop’s increasingly Activity-oriented workspace.

So what does that mean? Well, in gnome-shell, windows are organized by application – by the binary that spawns them. All your firefox windows are grouped together, all your okular windows grouped together, etc. You can still group windows on virtual desktops, but alt-tab is application-based, so as soon as you’ve got more windows than screen space you may as well have everything on one desktop.

In plasma-desktop, windows are organized by activity. This requires a bit of work on the user’s behalf (creating and switching between activities), but I have all windows for my odfkit work grouped together, all school windows grouped together, etc.

Obviously I think my way is better :) but I do have reasons for this, and I think it’s worth sharing them. I think that grouping by application is.. well, application-centric, application-developer-centric. It puts the branded applications – Firefox, Okular, etc. – front and center. This is easy to implement, and great for app devs’ egos, but I don’t think it serves the user very well. Why should they care that pdfs are opened in Okular, but jpegs are in Gwenview? (especially when some pdfs are just glorified images…)

Grouping by activity, in the KDE sense, is fundamentally user-centric. It puts the activity the user is doing – researching a paper, working on a project, reading comics, etc – first. The applications used to accomplish this activity are incidental, mere implementation details. The user might not even know their names, referring to them as “the web browser” or “the news reader”.

I’m a little doubtful about the gnome-shell assumption that applications shouldn’t be run more than once, too – but then, I’ve stopped using regular app launchers at all. These days I type stuff into krunner, usually a url… or dolphin, if I want to find a file… or I click links. I’ve set konqueror to open links in new windows, because activities make tabs almost obsolete. :) My regular apps stay open all the time – well, so long as that activity is open. :) Organizing my windows this way is *so* much nicer, I think I’d already go nuts if I had to give up using activities…

Anyways, since gnome’s choices seemed like big mistakes to me, I decided it would be polite to actually talk to them instead of just ranting in a blog. ;) I visited their irc channel for a few days, and although I don’t think I found any of the core gnome-shell people, I did find some people who were able to explain their motivation.

Apparently, they wanted to solve the problem of applications being started more than once when they’re not designed to. They do have their own gobject equivalent to KUniqueApplication, GApplication, but not all apps use it, so they’ve still got some apps where it’s possible to have two instances and Bad Things happen. So, they decided to solve it at the shell level by not launching more than one.

After doing my poll, this doesn’t sound quite so crazy; there *are* a lot of apps people only have open once. However, nearly everyone has that one app that they want two of… and in gnome-shell it’s up to the application to offer a way to do that.

Anyways, it may have been this one little decision that shaped the whole shell. Once applications were assumed to be unique, then switching windows *is* switching applications, and an application-oriented shell is the natural conclusion.

Also, I was wrong about them calling applications activities. They’re not calling anything activities, really. :) Their overview screen (a concept we toyed with in plasma actually) was called “Activities Overview” because activities means “what you are currently doing” – and then the string was too long for the UI, so they shortened it to “Activities”. I wish they’d shortened it to “Overview” instead – it would have saved a lot of confusion. :)

So… in the end, I still disagree with gnome-shell’s design, but I think I understand how they got there. It’ll be interesting to see how it’s received when it’s released, and how it evolves from there. After all, the world would be a pretty boring place if we all made the same decisions. :)

poll results

so I finally convinced myself to look at those poll results. :)
Not all of them, though; I got sick of it after about 60.
Given how varied (and sometimes ambiguous) the details were, it’s not very scientific anyways – some people talked about their systray, some didn’t; some counted their tabs, some just said there were “lots” – but I think I’ve still learned a few things.

The number of applications open ranged from 3-16, but most people had 6-7. The number of windows was usually similar; while it ranged from 3-38, it was usually 8-9 – two more than the number of apps. That means people are using tabs a lot, and rarely opening extra windows. There seemed to be three main resons for having extra windows: first, chat lists; second, okular (since it doesn’t do tabs); third, having different windows for different desktops.

The number of tabs is probably the most unreliable number; given a very liberal interpretation of “tab” (basically any form of switching views inside a single window – be it files, chat windows, or other resources) and counting one window as one tab (not zero), I counted from 3 to 90 – but I’m pretty sure it would have been more if some of the people who said “lots” and “many” had given numbers. :) quite a few people had >20 tabs in a single window – how do you guys *manage* that?? :)

Personally, I find myself moving *away* from tabs. It seems to me that they were invented to compensate for inflexible window management; now that I can sort my windows by activity, I have less need for tabbing. I open websites in new windows instead, only tabbing searches or sometimes links from the first page.
I still have a bajillion documents open in most of my kate sessions, though. I don’t think those would work so well as individual windows. :) although I do sometimes find myself trying to alt-tab to the previous file…

Anyways, back to the results: I was also surprised to learn that lots of people still minimize things to the systray. I’ve stopped doing that, myself… if I want something easily available, I put the window on all activities.

Oh, and while most of us pile lots of things on each desktop (well, I assume so – less than half the comments even mentioned desktops), some people have picked up a one-app-per-desktop workflow. Why? Is it because of the ability to assign a specific keyoard shortcut to each desktop?

That reminds me… on my screencast post, there were a fair number of people not understanding why we have both activities and virtual desktops (btw, thanks for being so polite and not going “omg u sux” – criticism is discouraging even when it’s done nicely). I think the poll shows one of the reasons: people have different ways of using virtual desktops. Some people use them in a similar way to activities, and – like the people who were happy with just icons on the desktop – won’t see a need for both. Indeed, they might be better off picking just one and not using the other. But there are other people, with other workflows, who will benefit from using both activities and virtual desktops. Perhaps we have an overabundance of choice here, but I think it’ll be a good thing, and will lead to lots of new ideas. :)

Oh, and I didn’t count browser usage in the poll, but if anyone feels like doing that I’d be interested to know the results. I seem to remember seeing several konq+FF comments, and several rekonq, but also some chrome, and one guy had at least three different browsers running…

…One more thing: Fuck ads. they sucked, they did *not* have the skip option that it claimed they’d have, so they’re gone. blip.tv owes me only $7 for that… and since they don’t pay out until you reach $25 I guess I’ll never get it. oh well :) it didn’t really seem worth subjecting everyone to 3-4 more ad-enabled screencasts.

Activities 4.6 screencast

Yay! Screencast! :)
For 4.6 I decided to do two separate screencasts: one intro to the basic features, and a separate one for cool advanced stuff. This is the intro, naturally; I’ll do the advanced one tomorrow or something. :)

…There’s another thing different here: I’ve enabled ads. I’m not sure about this – on the one hand ads are annoying, on the other hand a decent screencast takes hours to make and a bit of funding for my kde stuff would be useful. The ads haven’t started showing up yet anyways. (I guess it takes a while?) So, if you do see ads, tell me what you think of them.

Now on to the screencast:
[Edit: ads are off now]

Recent Activity… uh.. activities

:) There’s been lots of fun plasma stuff going on as feature freeze approaches.

Ivan’s been doing lots of stuff behind the scenes, making big changes to the activitymanager service. He’s also been polishing up the UI – you’ll notice that it highlights the current activity now (although the effect’s a bit subtle in whatever theme I’m using), various little bugs have been cleaned out, and it’s possible to edit the name and icon right there. :)

Then there’s the template thing Ivan mentioned – I hardly had to write any code for this really. The options are organized so that new users won’t be as overwhelmed (it was.. educational.. watching people try the UI at the opensuse conference) and anyone can write templates – little javascript snippets that set up widgets for an activity. :)

There’s one unfortunate thing here – the guy who was planning to actually create a set of templates for 4.6 is busy, and the freeze is thursday, and all we have right now is one ugly demo template I threw together in a few minutes. So if you want to create some – well, come pester me to document the process. ;) Hopefully I’ll have time to add a GHNS button to the UI too, and then more templates can be downloaded – but it’d be nice to have more than one shipped with 4.6. :)

That’s not all that’s been going on, though. :) Thanks to ivan’s changes under the hood, we can finally hook up my activity-session stuff to an actual UI! :D Clicking the little stop button in plasma now properly closes it, windows and all. :)
There’s more to do, like queuing things under the hood so that you don’t have to wait for one activity to finish shutting down before clicking hte stop button on the next; that’ll give us something to do after freeze. ;) I’m sure bugs will turn up too, now that I’m not the only person using the session thing. In fact, this will also expose bugs in most applications’ session handling. I already know that many qt apps (assistant, anki, etc) are not very good at session-restore; I’m still hoping there are a few things I can do on the kwin/ksmserver side, but a lot of this is up to the application itself.

One more thing: I created a dataengine for activities. This means that if you think you can do better than the current activity management UI, you can easily give it a try. :) There’s even examples to look at – I’ve ported the activitybar plasmoid and the switchactivity mouse plugin to use it. I haven’t added all the new API yet (you can add/remove activities but not open/close them) but I’ll get that done today or tomorrow. :)

One thing I’d love to see is a popupapplet for quickly switching activities. The activitybar takes too much space for my liking – I just want a cute little icon showing my current activity, and a popup that lets me switch to the others. then I could put it right into my regular panel. :) I’d write the applet myself if I only had the time…

btw, sorry for the lack of screenshots – my ksnapshot is crashing instantly this week. the joys of running trunk, eh? ;)
Maybe I’ll do a screencast instead, this weekend. :)

Activities fun [with screencast]

So… I wanted to spend this weekend offline. I did. I even logged out of irc.

But somehow I found myself hacking until 2 or 3am sunday night, instead. :)
I think it was worth it. ;)

The last few weekends I’ve been hacking on activity-related things, here and there. Fixing bugs, implementing things behind the scenes – last night I finally reached a point where the pieces began to come together. Today, I finally had something I could demo.

So without further ado, the demo:

(if the above doesn’t work, watch it on blip.tv)

I mentioned in there that it’s not done; we still need the activitymanager kded stuff to co-ordinate it (which ivan is working on :) and then plasma’s activity UI needs to use that API so that I can stop sending dbus commands. ;) It also needs a lot of code to handle edge cases, like properly restoring apps that are on >1 activity. The good news is, restoring the right activity associations for each window turns out to be way easier than expected. yay! :)

Oh, and there’s the taskbar integration to be done, too, and a million other little things… :) but we still have a month before feature freeze. 4.6 is gonna be awesome. :)

And now for something completely different: Berlin! it continues to be awesome too. :) I’ve got a book of Enid Blyton’s children’s stories in german, and I’ve been reading them with pete’s help; it’s a lot more fun to read ’cause the sentences are short and simple and the stories remind me of my childhood. :) I’ve also discovered that Dominion is a pretty cool game, and I can now speak german when I’m drunk. ;)

Living in another country has some funny effects. When we went to the mall today, I suddenly felt calm and happy to be there. I guess the atmosphere of blatant consumerism reminded me of Vancouver. ;) It’s weird how little things that aren’t really all that nice become happy reminders of your home country…

I still haven’t found HP sauce, though. :P Or black beans; they have every other kind of bean, but not my favourite! *sigh* On the other hand, I had no problems getting ripe avocados, and I’m still drooling over all the wonderful yoghurty things that I couldn’t get in Canada. :) And the cheese, and the cheap pizza, and the cheap beer, and, and…

a note on activity/plasma bugs

so, a bug snuck into beta 2. If you’ve never switched activities, you might not *have* a current activity, which can get you weird stuff like black screens. It’s fixed in trunk (as of a few days ago), but if you’re using the betas, go press meta-tab (the global shortcut for cycling through activities) to avoid this bug :)

Also, although the beta1 activity-creation bug was fixed, it gave some people a lot of useless activities that can’t be automagically cleaned up. If you don’t want to delete them all manually, here’s how to wipe out all your plasma and activity config:

1. log out of kde
2. log into a terminal
3. remove plasma-desktop-appletsrc and activitymanagerrc from ~/.kde4/share/config/ (might be .kde/ instead of .kde4/ on some distros)
4. log back into kde
5. do the meta-tab thing again if you’re on beta 2

remember: that will delete *all* your plasma config, both desktop and panels. you might want to back up the files first, in case you change your mind.

[edit] oh, one more note; if you’re using multiple screens or PVD, the migration from 4.4 to 4.5beta will make each containment a separate activity. so, if you had 4 virtual desktops and PVD turned on, you’ll have four activities, *each* with four containments. this might not be what you wanted. I had intended to add some code to try and create fewer activities during migration, but didn’t have time. it might still make it into 4.5; we’ll see.

status of activities in beta 2

so, good news: the activity bar and mouse plugin and so on are all safe to use in beta 2. :)
Thanks to neoclust, the main crash bug was also fixed. yay! :) I was seeing a second crash, but it was very odd and I can’t reproduce it any more, so hopefully that means someone else fixed it ;)

As for multiscreen – there’s still one bug to fix there. oy, and it seems I have some screensaver maintenance to do… I don’t think either of those will be trivial fixes, but I’ll try and tackle them soon. I’ve been busy lately – for the next three months I’m working on odfkit. :) It’s nice, but takes away a *lot* of my kde hacking time.

Added to that, I also installed opensuse last week, so I didn’t really have a kde dev. environment until yesterday. I’m still having some weird problems with 4.4 apps getting all broken in a trunk environment (somehow /usr/share/kde4 is getting totally ignored no matter what I do, if my PATH has trunk stuff in it, and kmail can’t get the imap protocol even if I run it in a 4.4 environment) but other than that it’s been surprisingly smooth. :) There were minor issues, of course, but most were easy to resolve, and now I have compositing and other such shiny things. :) I just wish I could figure out how to import my Amarok collection from the old hard drive…