Playing with Data in PAW

Auto-Update Again

2011-02-12T07:22:00.000-08:00

We have been using the Eclipse RCP framework as the basis for PAW. It comes with auto-update capability. However, as we started to use it, it becomes very complex- since as the developer, it's very hard to understand how the components are managed and when there's something wrong - what is wrong. We were also having to release changes as both updated installers as well as through the update mechanism for Eclipse. After a couple of issues that I spent weeks trying to debug and fix, I finally gave up on using the Eclipse P2 mechanism - it's too complicated, adds about 5-8MB to the download, and was making me spend more time on the framework than working on the product functionality.

Now, we're using the Eclipse OSGI mechanism to do installs and updates. It's much simpler to understand (the bundles are in clearly marked directories) and the API to make updates is much simpler.

TO INSTALL:
BundleContext.install('http://myserver/bundles/mybundle-1.0.0.jar')

TO UNINSTALL
bundle.uninstall()

The final program structure I came up with has a launcher which also comes with an installer. This runs with the OSGI bundle and some SWT libraries to show progress in a UI. It is basically something like this..

// create the framework
ServiceLoader loader = ServiceLoader.load(FrameworkFactory.class);
FrameworkFactory ff = loader.iterator().next();
Map config = new HashMap();
config.put("osgi.configuration.area", configDirectory);
config.put("osgi.user.area", workspaceDirectory);
config.put("osgi.instance.area", workspaceDirectory);
fwk = ff.newFramework(config);

// start it
fwk.start();

// do any installs and uninstalls
ctxt = frk.getBundleContext();
ctxt.install().....

// start the right bundle, then get the application and launch it
Runnable app = (Runnable)getService("myapp.id") // THIS IS PSEUDOCODE
app.run();

// finally stop the framework
fwk.stop()

That was roughly it to get the code working, and reduce the client footprint by 8MB - of Eclipse plugins that I wasn't using!

Cut'n'Paste matters

2009-11-30T16:44:00.000-08:00

The primary use case we've been targeting is where users want to build a repeatable process which involves pulling data from a reliable external source (it may need to be done again and again) and massaging it, perhaps w/ some other data, and then doing something w/ that data (loading it to a database, calling a web service, sending emails,...). Now our focus is shifting to how do we get users to be more productive once they have their data in PAW.

Filtering has been made simple, more than the auto-filter capability in spreadsheet tools. Choosing fields and changing their order was another must-have when you're analyzing data with lots of columns. And then finally, there's cut-n-paste.

Cut and paste is actually the primary mechanism users use to move data around, typically small bits of data, maybe used as a reference table for mapping or some departmental data needing slicing and dicing. We now allow the user to create a new table with the data or to paste it into an existing table.

First you paste your data and specify what table to load the data into. Then you map the fields to existing columns or specify new fields and their types.

And voila - you're done!

We've made the feature as simple as can be, but underlying it needs to ensure all the fields can be converted correctly. Pasting into an existing table is just as simple. I am looking forward to feedback on how this feature change what types of data you work with on PAW.

Technocrati, here we come

2009-07-17T07:13:00.001-07:00

eqtk34c7ry

Automatic Updates for PAW, finally!

2009-07-03T09:13:00.000-07:00

The web has ofcourse completely changed expectations for the feedback loop between customers and suppliers. In the software business, that means that you need to be able to turn around requests for features and bug fixes much much more quickly than 6 month or 1 year cycles.

For installed software, a key aspect of meeting this need is automatic updates. You're constantly getting bug fixes out and incrementally adding features based on user demand. The first requirement is the ability for the customer to very easily obtain an updated version of your product. Still a lot of customers will not bother to check for updates on a periodic basis. Bugs in earlier versions can be a huge stumbling block to people continuing to use your product. Without some sort of automated prompt to get an update installed, your customers are typically not seeing all the work your company is doing to better the product.

So, we just implemented automatic updates for PAW! It is seamless, it is robust, and it just works!

Automatic updates need to be a part of any platform for building software products. Eclipse RCP (Rich Client Platform) provides an excellent foundation for building software. With their latest release of Galileo, the automatic updates support is finally there with Equinox P2. We continue to be happy to have chosen this platform for building out PAW.

Performance Tuning for PAW UI

2009-03-12T07:19:00.000-07:00

As we near the final release stage for our version 1.0 product, the last few releases will be focused on performance. We have been tweaking the UI, so that the user gets into the product and is able to start using it easily. This caused us to move a lot of the functionality to a web-based paradigm - using browser controls for the menuing, displaying data, as well as performing certain actions (filtering, sorting).

Web-Based Data Display
So we decided to focus on the web-based data table as the primary go-forward option. We made it the default for showing large sets of data resulting from running processes. It was already the default for tables that one can edit, but processing results tend to be much larger and hence it is imperative that the user can see and navigate through the data quickly. Hence the focus on performance.

In terms of performance enhancements, we reduced the time to display large datasets from 50+ seconds for 3k+ rows (when using the web-based rich table) to 7-8 seconds. This is inline with the native windows table w/o the rich functionality (multiple lines in a table, filtering, etc..).

This was accomplished using DocumentFragment for DOM manipulation so that we are not causing the browser to re-flow the document when each cell is added. Many thanks to John Resig (http://ejohn.org/blog/dom-documentfragments/) for this helpful hint. It made a almost 10x improvement, beyond the 2-3x improvement that's been noticed for smaller amounts of DOM manipulation.

Interactive UI
Process runs have also been moved to being "jobs" which run outside of the UI and then update the UI once the data has been refreshed. This allows the user to continue to perform other actions. Again, this makes the application more responsive to user input. Similarly, we have added a preview when importing data from the web, allowing the user to see where a page might have changed in case the web import isn't working as expected. This is also provided in a separate window.

As next steps, we will be looking at tuning some of the more resource intensive operations like merging large datasets through the use of more advanced algorithms and more specialized use of RAM.

Providing simple and advanced options

2009-03-02T06:46:00.000-08:00

It has been a long time since I've posted. Been hard at work looking at understanding our users and what issues they might be having - user research. One of the things that we found with software as powerful as ours, is that people need an easy starting point. Imagine if you saw Excel for the first time and were trying to figure out what to do.

So we've been busy adding more user-focused questions through pop-ups asking users if they would like X or Y ("would you like to re-run your process now?"). Most importantly, we've added a user guide view which distills the essence of what PAW enables and provides a focused task pane for just those functions. We feel so positive about this that we've made it the default starting view when users start the application!

Existing advanced view:

And the new simpler user guided view:

Please send us your feedback as we try to make further improvements!

The Perfect Match

2008-08-26T21:05:00.000-07:00

Today at work I was interviewing some candidates for a position we have been trying to fill for quite some time now but haven't been able to find a good match. This experience got me thinking can PAW be a "perfect match"?.

Today was also billing day, the two days in the month when I am drowning in numbers, pulling multiple time sheets for all my direct reports working with multiple excel worksheets. In these two days I will have worked over 10-12 hours per day pulling reports, summarizing them, matching and merging finally building out my worksheets to be sent to my finance department for billing.

Month after month there is all this manual labor involved in putting together these workbooks. This prompted me to think of getting a software product that would help reduce it. It would be so nice to get back 4-6 hours. A 50% in time saving ---> SWEET. I opened PAW started to build my monthly time reporting worksheet from scratch. It took me about an hour to set-up and automate the process the way I wanted it. I tested it and it worked. I wondered if there was something out there that could do everything I had in my head, PAW only addresses 85% of the steps I envisioned.

On the drive back home I realized that next month I will only have to do 15% of the work as PAW will automatically do the balance 85% and export the excel file to me, that made me so happy. In today's world of instant gratification where software only gets cheaper, faster, better one has to realize "the perfect match" is also changing constantly, whats in today is out tomorrow, whats considered hot today is dead tomorrow. I now look at PAW in a new light my PERFECT MATCH light my solution for today, tomorrow will bring something that is faster cheaper better. Until then I have found my PERFECT MATCH.

Marketing Challenge

2008-08-20T23:41:00.000-07:00

I have been working in the field of marketing for quite a few years now. Most of my job involves interaction with data in some level or another. Some things I regularly working on are - getting data files of customers for my next email campaign, status of payments received from a particular client, or simply an export of all the responses from my marketing survey.

Every time I get data I have to spend time doing some processing to it so that I can visualize it in a fashion that is understandable to me. Today I was faced with such a situation and PAW came to my rescue.

Today, I received twelve files of data exported in excel from a vendor who implemented one of my customer survey's. All of the raw data I collected was important to me however in addition to this raw data there was more value in analyzing the following -

1. How many of my customers completed more than one survey?
2. What is the number of new customers to the email file vs. my control?
3. Isolate email addresses that have subscribed to my monthly newsletter
4. Isolate the email addresses that have shown an interest in my next webinar

Traditionally, I had to have someone from my team or the database manager right some SQL query's to parse this information and send it to my team. In the PAW world today, I or some one from my team pulls all the information in PAW and withing the hour has all the various data files ready to send to the various teams.

Merging or de-duping data takes minutes in PAW even for large files (each of my files had over 35k rows in it) and what's even better is that I don't have wait for other teams to send me the parsed data. My email marketing campaigns are more releveant as they are sent out in a timely fashion bringing more ROI and revenue to PAW.

Automating your daily work - dealing with web logs

2008-07-18T21:58:00.000-07:00

For techies, there's many many tools to automate things. Lots of scripting languages, multiple open source libraries, articles and tutorials on sites like IBM developerworks & ServerSide. All that's required is an inclination to automate and time.

For more of a business user, there's not really a tool that makes it easy to automate. You can either find a service that solves your problem, hire someone to write some scripts and then keep paying them to make updates when things change, do the thing less often but in the good old-fashioned manual way.

PAW is intended to help the business user automate their work... the repetitive part. You can download files from a server (there's a tutorial on this), you can import and parse the file, find the specific data points, and then summarize it. If necessary, you can combine your web log data with information in your customer database. Things like the # of page hits in a session measured against when they registered on your website.

Best of all, you do small bits in PAW and see the results right away. If something doesn't work, you know right away and can fix it. You're working on your data, kind of like what you do in a spreadsheet.

We would love to get feedback on whether this mix of being able to automate and see your results quickly is the right blend of immediacy, automation, and ad-hoc processing.

Bookmarking data

2008-07-10T06:36:00.000-07:00

When the browser was introduced, one of the first innovations to really take off was the bookmark. We got used to bookmarking pages containing useful information... Similarly, very often, in the work environment, we are trying to keep on top of a few critical pieces of information. It might be the project budget, sales to date for my region, or updates to certain bugs that I need to re-test. Our goal is to allow the user to link to that piece of information and bring it into the user's context, rather than the user needing to open up the application or web page and then navigate to find that piece of data.

With the PAW tool, you can define the process to get to those snippets of information, in some cases calculated. Then you can very easily get updates for that data in the PAW environment. In essence, you've bookmarked that data!