SAINT toolkit

Release!

2011-06-14T08:36:00.001-07:00

Yes, it is finally there: a new release of SAINT!

The version I have put online is version 2011.01 beta 1. It contains all the basic tools, and supports both Access and MySql as database backends. Many issues were fixed, but it is still a beta. Please let me know of any problems, especially of regressions.

Get it while it is fresh!

Saint as Qt Ambassador

2011-03-15T01:31:00.001-07:00

Recently, SAINT was added as a Nokia Qt Ambassador program. That means that it being used as a showcase for the capabilities of Qt and the different areas Qt is used in.
Check SAINT on as a Qt showcase here!

Database layer working: ISI parser to proof it

2011-03-14T08:31:00.001-07:00

Moving away from Access as the only database backend supported by SAINT has been on the TODO list for quite a long time. As it turned out, creating a system to allow for multiple database backends was not all that easy. It required changes on many levels, which I'll detail a bit below. But don't let that spoil the big news: it works!

Yes, that's right, you can now actually use different databases than just Access with SAINT. At this moment, I have it working with just the ISI parser, but that is already a very important step. I use it as a testbed for all the other tools, which basically need the same infrastructure that ISI parser does, though usually to a lesser extend. That comes down to: if it works for the ISI parser, it should (mostly) work with the rest of the tools too. I will release a beta to demonstrate the working ISI parser as soon as possible.

Nitty-gritty stuff
So... what kind of changes were needed in the SAINT software stack to get all this work? Quite a lot, as it turned out.

Driver backend
The first thing that needed to be done, was to come up with a driver backend that would support all the operations we need on databases and that will work for all of them. The standard Qt database drivers only support abstractions for manipulating the data (Data Manipulation Language or DML), not the data structure (Data Definition Language or DDL). Since SAINT manipulates the data structure all the time (creating tables to store results, for instance), we need support for all of these operations. But: not all databases do that in the same way! SQL is a standard only to point... To further complicate things, not all database support the same data types, nor do these types map to the datatypes used inside SAINT and Qt in a trivial way. To store a value like a date or a long text string, you need different data types in Access and MySql, for instance.

I choose to extend the standard Qt Sql database drivers with additional methods, and in that way create new drivers that can be used with the standard Qt database system, but that offer support for far more operations. Currently, I have drivers implemented for Access (on top of the Qt ODBC driver) and MySql (using the standard Qt MySql driver).

Abstracting database access inside the tools
In order to able to actually use these shiny new drivers, all access to the database needed to be abstracted out of the code. That is: nowhere in the code could direct SQL be used anymore, nor could I use things like type names in the form as strings any more. All SQL statements needed to be generated by the current driver based on a common format. I used Qt's own QSqlRecord for that, in combination with a hugely extended list of supported query types for QSqlDriver::sqlStatement(). Where this last statement is barely documented in the Qt documentation, is turns out to be essential if you want to make your application database agnostic.

User interface
Creating a user interface to select a file on your local file system is not all that complicated. But coming up with a good interface to select or create a generic database, either as a file or running on a server, is a different matter. It took quite a bit of effort to design a build a UI that seems to be easy enough to use, and that can be embedded into the different tools with ease. It will probably need some more tweaking, but here is an idea of the current UI:

The widget that can be embedded in the different tools to select the database:

It features a drop down box with the 10 most recently used databases (with an icon representing the type of database), and a button on the right to open up a dialog box that allows to select another database currently not in the list. This widget will allow drag & drop in the near future, so you can simply drop your exising database file or .sdbd file (more on that later) on it.

The Database Selection Dialog that you get when clicking the button:

This dialog lists the databases that are known to SAINT. For each of them, a small file is stored in your documents directory (under SAINT/database descriptions/, to be exact) containing all the information that is needed to connect with the database, along with some meta information. You can even send these files to your college to easily share access to a server-based database if you want.

Select or create another database:

This is the page that allows you to select or create a database that has not been used before. The drop down box on the top displays a list with available database types. Currently on Windows, you can choose from Microsoft Access and MySql. The rest of the dialog will adapt itself accordingly. Showing on this screenshot is what you get when selecting an Access database.

Dialog for MySql databases:

The above displays the same dialog, but with the MySql database type selected. Once you enter the relevant data on the server and your credentials, available databases on the server will automatically be displayed.

All database types will insist on an Alias to be created for the database. This alias is the name for the database that you defined by selecting a file name, or by entering a server address, port, user credentials and database name. It is used in the file name for the .sdbd (Saint DataBase Description), in the drop down for the database selector widget and in the list of existing databases. A suitable alias will be suggested automatically.

Word stemming

2010-03-30T04:55:00.001-07:00

One of the issues with working with (bibliometric) data is data cleaning. While we already have an experimental tool called Record Grouper that is being refactored at the moment, we also needed a quick-and-dirty way to do some basic cleaning on words.

To facilitate this need, I have added a new feature to the Word Splitter tool. The tool now builds an additional table called Wordstems, and it adds an additional field to the Words column with a reference to a record in this Wordstems table. Each word is now being stemmed by using the Porter stemming algorithm. I am looking into replacing this algorithm by another, more accurate one, but the idea will stay the same if that happens.

So now, each word will get it's stem (according to this Porter algorithm) associated with it using the stem's ID number. This makes it easy to treat words that have the same stem as the same word, so we can get rid of the difference between "robot", "robots", "robotic" and "robotics" easily if we want.

On a note that relates to the last posting on database compatibilities. While the Word Splitter still only works with MS Access, under the hood a lot has changed. I have implemented a driver system with extends the database drivers that are available in Qt by default, and works around a couple of bugs on the side. Based on this, I have created an Access Driver that is now being used by the Word Splitter tool. While it is not completely database independent yet, it is a good start and a good test for the new driver.

Database compatabilities

2010-02-24T02:09:00.000-08:00

At the moment, SAINT only supports Microsoft Access as a database to store your data. Because most of the tools are quite tightly bound to accessing this data structure, you are really bound to Access. That is really becoming a problem.

First of all, Access itself is a problem. Access has some limitations that limit the amount of data that it can handle to about 2GB. However, that limit includes temporary space needed for the execution of some complex queries. In practice, you will thus run into this limit much earlier in the form of a vague error message. Access has other problems too, but this is the main issue. Access does have two big strengths: first of all, it is easy to handle it's databases. You can just copy over files, and it is already installed in many professional environments. Second, it has a quite good visual query designer, making Access less of a database than it is an analysis environment in the context of using it with SAINT.

The second problem with Access has to do with Qt, the toolkit that is used as the base of SAINT. Qt supplies a set of database drivers, that makes it possible to interface with different databases. It does not have a driver for Access/JET however. SAINT accesses the databases using a driver for the ODBC layer. That has an impact on the speed of the connection, as well as on the supported features. A solution would be of course to write a database driver for Qt myself, but that is not a trivial task.

The third problem is that of cross platform compatibility. Access only runs on Microsoft Windows, but in the scientific world, macs and linux machines are not all that uncommon. It would be very good if SAINT would be able to work on those systems as well. Because Qt itself is cross platform, that is not so hard, were it not that SAINT is bound (too) tightly with Access and thus with Windows.

So, it was time to come up with a solution.
Like I said, Qt supports database drivers for different databases. These work cross platform. However, they don't entirely abstract away the differences between the databases themselves. Different databases support, for instance, different data types. They also use a slightly different query languages. The base is the same (SQL), but the dialects differ. That is easy to understand: the databases are different for a reason, and they support different features. That also means that you sometimes have to talk to them in a slightly different way. Qt helps with this, but not enough.

This issue makes it hard to simply exchange one database for another. That would also create other problems for the users. What to do with existing data sets that are in Access already? An optimal solution would be to build in support for several data back-ends, and let the user choose in an easy way which one to use. This way, the tools can be used for small, simple sets that work perfectly on Access or some other small local database, but also on huge sets that run on a remote server.

To achieve that, I need to extend Qt's abstraction from the database back-end. Currently, it supports basic manipulation and querying of the data in the database, but it does not support changing the database structure. That feature is needed in many tools though. Many parts of SAINT create new tables, or extend existing ones. The solution I have decided on focuses on extending the Qt SQL driver model to support these operations, and then re-factoring the existing SAINT code to use this abstraction and to remove the current explicit SQL code that is used everywhere. All SQL will then be generated by the abstraction layer, making it possible to support different databases as a back-end for SAINT by just replacing the database driver. Of course, there is also some work to be done in the interface, as connecting to a remote database over the Internet needs a different setup than just pointing to a file on your local file system.

I am developing this code in a separate branch of the code for now. It will be for SAINT version 2.0. So far, the progress is promising, but it is still far from production ready. Expect a first beta release of this code somewhere in April.

Changes in ISI data importer: increased compatibility with other sources

2010-02-10T00:56:00.000-08:00

Recently, I was contacted by somebody who tried to use SAINT on a dataset converted from PubMed to the ISI text format. That did not work. It turns out that the ISI data importer relied on the existence of a really ISI specific field to ensure records were unique. That field does not exist in the data the converter tool created, and thus the application regarded all of them as the same: the code was empty for all of them.

A small update fixed that. Now, instead of only relying on the ISI code field, the program will try other options. The first option tried is to see if there is a valid DOI code. That code is unique too, if it is available. If it is, that code will be used as the unique identifier. If there is no DOI code, an artificial code will be generated based on the journal name, the ISSN number, the year, the full name of the first author and the title of the article. That should yield a pretty unique identifier. The code will be prepended with an identifier that tells you where the code came from. These can be "isi", "doi" or "saint".

As a side-effect of this change, I have also added a new field in the articles table output. The DOI field is now included. That also creates an interesting matching opportunity with the cited references.

All very nice perhaps, but... The changes can lead to two problems:
1) If you import data from multiple sources, you may run into double values. This can be the case if an article occurs for instance in both an ISI and another source. In the ISI source, there will be an ISI identifier, so that will be used as the unique code, even if there is a DOI field too. If the same article appears in another data source that does not have the ISI identifier, we'll run into a problem. The DOI field may be used as the identifier, and the articles are no longer identified as the same one.

2) If you use this newest version of the ISI data importer to augment data from an earlier version, old articles will not be recognized as the same anymore, because in the previous version, there was no "isi" prefix to the code. If you plan to do that, you should append the string "isi:" (w/o the quotes) before every code field. You can do that by running the query below:

UPDATE Articles SET Articles.code = "isi:"+[Articles].[code];

ISI importer demonstration

2010-01-25T07:13:00.000-08:00

After some good responses on my video demonstrating the Matrix Builder tool, I decided to make a video demonstrating the ISI Data Importer tool. Again, the video is available at Youtube.

The video demonstrates the complete process from selecting your input file to starting the actual parsing, and highlights the new features of the parser that help you identify the next step. I hope it will be useful to you!

Update (January 27, 2010):
OK, I am still learning how to deal with Youtube, and how to optimize the videos properly. I have changed the video for an HD version. If all goes well, that should work now. Watch in full screen at 720p quality for optimal viewing. It may take a little time for Youtube to process the video though.

Video demonstration

2010-01-15T07:28:00.000-08:00

I did a first attempt at creating a demonstration video for our SAINT tools. The first tool to get it, is the Matrix Builder tool. It's very basic, and runs for less than a minute and a half. I will create more elaborate videos later on, but it may be useful for somebody already.

Update: the video has been replaced with a higher resolution version. This should improve readability quite a lot. It may take a little while until the HD version is actually updated and you'll notice the improved quality.

Wrong file in download link

2010-01-15T01:58:00.000-08:00

When uploading all the files, and updating the CMS to point to them, I made a mistake. The installer I file I pointed to from our CMS at the Rathenau Website was actually a very old installer. The installer pointed to from this blog was the correct one.

The mistake has been fixed now, but still: embarrassing. Sorry for any confusion or inconvenience!

Release 2010.01 beta 1 is out!

2010-01-13T00:45:00.000-08:00

Yes, I finally made a new release ready for testing.

This new release fixes several bugs, the one bugging you most probably being the ISI parser issue. Even though there was a work-around, it was an annoying issue. Sorry about that!

Another big thing in this release is that it is the first release to include the community detection. As part of the Network Tools program, you can now employ the algorithm described in V. Blondel, J.-L. Guillaume, R. Lambiotte and E. Lefebvre 2008, Fast unfolding of community hierarchies in large networks. More on that algorithm you can find on their website.

As already described in the last blog posting, there are also some changes under the hood of the Matrix Builder tool. Besides labels and edge values, you can now also output other properties to the Pajek .net output file, allowing you to display more aspects at the same time in your network.

There are many other small tweaks and bugfixes in this version. Expect more additions before the final release. When that will be released? When it is done!

Download the latest beta toolkit (installer for Microsoft Windows) here.

Big changes in Matrix Builder

2009-12-09T03:46:00.001-08:00

More news on updated tools: the Matrix Builder.

While simple in concept, the Matrix Builder is a very important tool in our toolbox. And it has seen some important changes! Let me take you through them.

Just like the ISI parser, the matrix compiler has had some subtle interface changes. The window can now be minimized, and the close button has been removed. The old OK button has been renamed to Run. This will result in less confusion.

More importantly, is the introduction of properties. That is: more data can be attached to the output than just connection strengths and vertex labels. There are many such properties possible, and we have a wish list for almost all of them. The basics are implemented, and all that remains now is to implement more 'plugins' to support more properties.

Quite a few nice properties have already been implemented. The most useful will probably be the vertex size property (scale node sizes according to some value), and the vertex colouring.

The new version will bear version number 1.3

ISI parser updates

2009-12-09T03:29:00.001-08:00

It has been a long time since my last blog. That does not mean that developments have stopped! I'll try to fill you in on what happened in the meantime. I'll start with the oldest tool in the box: the ISI data importer.

Recently, I made an update for the ISI data importer. Most changes are pretty small, but still...
The first issue that was addressed, is a limitation that existed with selecting files. If you selected lots of files, especially if they had long file names in a deep level of you directory hierarchy, it could happen that you'd run over 32 thousand characters for the file names. That resulted in leaving some files out, without any warning! The issue was addressed by changing the way multiple selected files are displayed in the file selection widget. Instead of just listing all the file names including their paths (which you will never read anyway for 50+ files), you now get a listing like

'file_1' and 999 other files in directory ''file_0.txt' and 999 other files in directory 'C:/Documents and Settings/andre/Desktop/data'

Much more readable. The way the display is formatted, depends on the number of files selected.

Another issue that was addressed, is the laggy display of the progress. If you hid or obscured the progress window during a parsing operation, it would take untill the next new file untill it was updated again. This is now fixed. What's more, the window can be minimized, and the main window is hidden during the parsing process. The progress bars are also changed. The files progress bar has been replaced by a write progress bar, that displays the amount of parsed records (articles) being written to the database.

A new feature has also been introduced: it is now possible to add data to an existing database! That means that if you select an existing database as your output file, you are now forced to choose what to do with that. You can either augment the existing data (no duplicates will be made) or you can overwrite the existing database completely.

A user interface change will enforce that you actually have make a choice: The OK button has been renamed in a Run button, and the Close button has been removed. Closing is done by just closing the window. The Run button will only be available if no problems have been detected. If there are problems that prevent running, hovering your mouse over a small warning-sign icon will tell you about what they are.

This new version of the ISI data importer will get version number 1.2.

Progress with Relation Calculator

2009-08-06T08:50:00.000-07:00

The relation calculator tool that I introduced in my last blog, is shaping up nicely. Sure, it is not all smooth sailing, but it is really starting to look like something that could be very useful indeed. Let's use a screenshot of the current version to illustrate where things are heading:

What do you see?
The main thing you'll notice is the area on the right, where you see boxes that are connected by lines. Each one of these boxes represents one simple step in the process of doing an analysis. In this case: calculating a Jaccard coefficient based on a co-word/cited reference combinations between journal papers. A nice, basic selection of such basic steps - components in the terminology of the application - is already there, though not fully implemented yet. The list of components can be extended later on using plugins. The available components are visible in the list on the top left.

Each of the components have inputs on their left and/or outputs on their right. Outputs can be connected to one or more inputs for other components, thus creating a graph. Note that no circular connections are allowed. The user can use drag & drop to put components on the screen, and to connect inputs and outputs together.

So, how do these separate components become an analysis?
On the bottom left of the screen, you can see the Execution order window. Here you can see the order in which the components will be executed. This is determined by their connections, the question if they present an interface at run time, and their positions on the screen. You can open and save analysis sequences for reuse and distribution.

Once you have hooked up every component in the right order, you can run it. You will be presented with a wizard-type interface that guides you through the UI elements that each of the components presents (if any), and that present information about the progress of your analysis if there are long run times involved.

Building a new tool: Relation Calculator

2009-07-17T03:20:00.000-07:00

As announced in the previous posting about development prioritites, I will be working on a tool to calculate scores for relations between objects in the database. That sounds very general, and it actually is. Allow me to elaborate...

background
The basic way of creating relations in our relational-database based data store is to create SQL queries that express the relation (e.g. correlation or distance measure) you are after. You can for instance relatively easily express a bibliometric coupling measure in SQL. It is also possible, but already more complicated, to express a Jaccard index over, say, title word similarities between articles in your database.

This approach, while very flexible in theory, still has some limitations in actual practice. The first is user-related: not every researcher wanting to do this kinds of analysis is a hero in creating SQL queries. That limits the usefulness of the tool set, not because what people want can not be done, but because it has a too high learning curve to actually do so. Providing standard database views for standard analyzes only solves this to a limited level.

Another limitation is that database engines are not always as efficient as they could be in evaluating the expressions that you need to construct often used measures. Also, they tend to use only a single thread to do a single query, thus making limited use of the resources of our modern multi-core computers. Using GPGPU techniques to speed up calculations is completely out of the scope of SQL for the foreseeable future. All this means that our calculations take a lot longer then they need to take, and sometimes run into arbitrary limits that they need not run into.

As stated above, providing standard views can only work up to a point. We want to be able to calculate relationships between all kinds of items in the database (articles, journals, authors, ...), and we want to be able to use different measures for them as well (Jaccard, cosine, Salton, ...). Providing standard solutions for all of them is simply not doable. It would result in an exponential increase of pre-defined views for everything you want to add, and would basically create a mess in what is embedded in the database. That's not a very inviting prospect. What we need is something that is flexible enough to allow calculating relations between all kinds of items in all kinds of ways we can think of.

So, we need a tool that can make these calculations:

more easy to use (at least for the standard analyzes), and
faster to compute, while still
be as flexible as possible.

introducing Relation Calculator
The Relation Calculator tool should become that tool for you. It uses a concept of building blocks, that can be put together to create a calculation that makes sense. Each building block will be responsible for a small part of the chain, like selecting which database to operate on, selecting views and fields, loading the data from the database, calculating a Jaccard index, etc. Each building block has inputs and/or outputs, that can be connected to each other. In this way, a calculation for a relation can be defined. Some building blocks will present a UI to the user during execution of the calculation, for instance to allow selecting a database or some parameter like a threshold. Other building blocks will just perform some service, or maybe even one simple logical operation. Of course, these configurations of blocks can be loaded and saved, to be re-used later on. A set of pre-defined configurations can then be presented to the user.

New building blocks can be added as plugins later on, making the tool extensible. Another option to extend the functionality is to use building blocks that execute a script as their payload. For instance, you would be able to define a function in JavaScript that expresses a relationship between two authors based on some input data. That script can then be used in a configuration. The possibilities are virtually endless.

I envision a graphical environment where a user would be able to drag and drop building blocks on a canvas and graphically connect the input and outputs of the blocks. This would create a very simple way to define new configurations to calculate new relations.

relation to existing code
There already is some code that does the kind of work I described here. Like I mentioned in my previous blog, the current Record Grouper basically calculates such relations already. There is also already some code available that lays the basis for a script-based relation calculator. These existing pieces will of course not be just thrown away. They will have to be refactored to be able to use them as building blocks in the new Relation Calculator.

status
I am now very bussy implementing the infrastructure for all this. Though it is a lot of work, I am confident that it will work. Some details still have to be filled in, but I don't expect major obstacles in the near future. I hope to get a basic model working soon.

The idea is that

Development priorities

2009-06-30T02:38:00.000-07:00

We have set some development priorities after the first release that we did recently. These are partly reflected in the Ticket tracker, but to be honest that doesn't quite do it justice yet.

A small but important update is to change the output format for the Matrix Compiler tool. Currently, it outputs to a full matrix in DL format. This is a bit inflexible, as it does not allow attributes other than a label for the nodes, and it outputs a full matrix even for a large, sparce matrix. That results in bigger output files, and thus longer processing/io time. In the (hopefull near) future, it will allow for the addition of including attributes to nodes as well as connections. You can track the progress on this issue here. We're already testing it...

A larger task has to do with reworking the Record Grouper and the Relation Calculator. The first part of that job is to specialize the Record Grouper to do only that: group objects based on some kind of relation between them. This means ripping out a large piece of complicated code (but not throwing that away, see later) and focus on a good UI to make it easier to work with. This means fine tuning the layout, but also add options to undo, store and re-load your work, etc. That is a lot of work in itself, but it will simplify the code considderably making it easier to maintain.

Another large task is to get the Relation Calculator into a usable shape. This is a complex tool. The basic idea is that will become a specilized tool to calculate a similarity or distance measure between any two objects in the database, and be as flexible as possible as to how to calculate that. Currently, we only use SQL queries to calculate such scores, but that is sometimes limited, often complex, and usually relatively slow because most SQL backends don't use multi-threading for single queries, let alone utilize things like letting the videocard do work for you.

You can express a lot of such measures in SQL, but it is often complex to do, especially if you are not that used to using databases. That makes the current way of working harder to use for novices, but also for seasoned researchers who just are not that into SQL. It is however more flexible than being stuck to defaults too much. In this tool, we want to make the standard things easy, and yet be as flexible as possible to enable more advanced use.

The goal is to make the standard analysis that you would normally run on a database generated by the ISI Data Importer and processed by the Word Splitter and optionally the Record Grouper as easy as selecting them and optionally set some time slices or threshold, after which ready-to-analyze output is generated. We aim to release a first working combination of these tools at the end of August.

New name: SAINT

2009-06-29T04:06:00.000-07:00

We did it: we came up with a nice acronym as a name for the toolkit. Since Science Research Tools is a bit generic, and SciSA toolkit too much linked to our department name, we came up with a new name: SAINT. That is an acronym for Science Assessment (or Analysis, at your choice) Integrated Network Toolkit. This name will be used everywhere to refer to the toolkit from now on, though it will take a bit of time for it to be used everywhere consistently. Note that our URL's will not change, so there is no need to update your bookmarks.

SAINT will save you (time).
That's just one of the many possible catchphrases of course... We'll come up with some new ones in due time to advertise the toolkit to the outside world. If you have any suggestions, please let us know!

Mini-demo during coffee breaks

2009-06-26T01:55:00.000-07:00

The official demonstration session on the e-Social science conference today is scheduled for the last 20 minutes of the lunch break. Because of the nature of the lunches (a three course affair that seems to run over the allotted time every time so far), I have decided to cancel this demonstration. Experience by others so far are that no-one shows up for these sessions.

For those interested, I will be giving mini demonstrations with my laptop only in the coffee breaks in the morning and/or afternoon. Simply find me, and ask! I am exited to fold open my laptop for you to show you our work.

Introducing: Matrix compiler

2009-06-26T00:07:00.000-07:00

In this third instalment of the 'Introducing...' series, I will be talking about the Matrix Compiler tool. While you can already see a lot of interesting things by just looking at the tables and views in the database that contains your data, visualization can be a huge benefit to recognize patterns. That means that you will need to somehow transform your data out of the database into a format that you can use for visualization.

The Matrix Compiler is a tool that can do this. It can use the database and translate it's data into a format that you can read into Pajek, a well known visualization and network analysis package.

Analysing and visualizing networks means that you will be dealing with two kinds of things. First, there are the objects that are connected, which we^* will call the nodes. Next, there are the connections between those nodes themselves. At least the latter should be available as a table or query/view in your database. For briefness, I will call them all a view from here. Depending on how you build up your database, it is possible that your connections view will contain complete labels (or other attributes), but it very possibly may only contain an ID of the node in the database, and the label for the node is defined in some other view. The Matrix Compiler supports both modes of operation: the labels for the nodes can be retrieved from either the connections view, or be looked up from an external view.

After opening the database and indicating the name of the output file, the matrix can be set up. When constructing the matrix, you start with selecting the connection view, by placing the cursor in the corresponding box and either typing the name or selecting it from the view by double clicking on it. You then select which field in that view represents, respectively, the value for the relationship, the row and the column. A connection view thus needs to have at least three fields: two to represent the nodes you are connecting, and one to indicate the strength of that connection.

The next step is to define the structure of your matrix. There are many options for that. If the types of objects in the rows and columns are the same, you may want to create a square matrix where all the nodes appear as both a row and a column. This makes sense if you are, for instance, constructing a matrix to represent co-authorships, but not if you want to display in which journals authors publish.

If you chose to create a square matrix, you may also choose if you want the matrix to be symmetric or not. In a symmetric matrix, the value of M(a,b) would be identical to M(b,a). Again, in case of co-authorships the meaning of author a sharing a co-authorship with author b is the same as saying that author b is sharing a co-authorship with author a. But for a citation relationship, a citation from a to b is different from one from b to a.
In both cases, you can also choose if you want to set the diagonal, that is M(a,a) to a set value, or if you want to use values occuring in your data. This can make sense to filter out things like self-citations from your data.
If your data contains data for the same relation more than once, you can choose how to deal with that. The options are to use the first occurrence, use the last, add the occurring values or multiply all the occurring values.

The last step of creating your matrix, is to choose where the labels for the nodes should come from. As explained above, there are two basic options. If the correlation view already contains the labels, just select the appropriate option and you are done. If that view contains references to nodes though, you can now select where to get the actual labels. To continue our example on the co-authorships, it would make sense to select the Authors table as the table to find the labels, and use the full author name as the label for your nodes in the network.

Note that if you use an external source of labels, you can choose how to deal with nodes in your label view that do not appear in the correlation view. For instance, authors in your Authors table may not have any co-authorships. That means that they will not show up in your correlation view. You may or may not want to include these unconnected nodes. The choise is yours.

Note that outputting very big matrix files, can take some time, as the output size is O(n²). We are planning to change the output format shortly, from a matrix form to a list form. That will result in smaller output files for big matrices, and will also allow the inclusion of attributes other than a label to both the nodes and the connections.

* Pajek itself uses a different terminology. It instead talks about Vertices for the nodes, and Arcs and Edges for the connections between these nodes.

Introducing: Word Splitter

2009-06-24T21:43:00.000-07:00

In the second installment of the "Introducing" series, I will tell you something about our small utility called Word Splitter. The idea of the word splitter is simple. You point it to a field in your database that contains text, like a title or an abstract. The word splitter then builds up a table with all the words that occur in the database in that field, and a table with pointers between the record identifier that the text came from and the record identifier in the words table, so you can find which words belong to which text-record. To make it possible to re-construct the text based on the sentence, the position the word was in is also stored in that pointer table.

Our tool can do more than that though. First of all, it can process multiple text fields from your database at the same time, making it more efficient to work with for you. So, you can split both that title as well as that abstract simultaniously.

Furthermore, the Word Splitter can use stopwords. Stopwords are, in their basic form, lists of words that are ignored when splitting the text, for instance because they are too common. That means that if you use stopwords, not all words from the text will be stored in the Words table, nor will pointers occur in the couple table. However, for different purposes, you may want to think of the procedure in two ways. One option is to first split the complete text, then remove the stopwords, and then store the words and their positions after removing the stopwords to the database. This will result in consecutive word positions in the couple table, even if there used to be one or more stopwords between two words in the original text. Alternatively, you can split the complete text, note each word's position, remove the stopwords and only then store them into the database. This will result in word positions that reflect the original position of the word in the text, but leaves them non-consecutive.
To provide maximum flexibility in the analysis, both these positions are stored in the couple table.

The Word Splitter can use several stop word lists at once, and furthermore knows three kinds of lists. First, it can use simple text files that contain lists of words. Second, it can use a field in an existing database that also contains a list of words. And last, it can use lists of regular expressions. These expressions are patterns that each word is matched against, and if it matches, it is regarded as a stop word. That allows you to, for instance, filter out numbers or dates without having to write them all out.

To make it easy to use these stop word lists, you can create sets of these lists, and store such a set as a file again. This way, you can easily review the stop words you used for your analysis, and you can re-use the same set later on. You can also use one such a set as your default set of stop words.

This was a basic introduction to our Word Splitter tool. I hope you will like using it!

Installer for windows online

2009-06-24T12:43:00.000-07:00

With the official launch of the toolkit only a night away, I have just uploaded an installer for the Windows platform to the file storage we have on Assembla. Of course, our website has been updated to reflect that. Other files you can find there include documentation, but also testcases to reproduce bugs.

Tomorrow at 11 AM, I will do the first demonstration (note, the time has changed from the earlier announcement) on the e-Social Science conference in Cologne. I will give a little bit of background, and then quickly move on to actually showing the attendants the tools on some real life data. Of course, I will also show some pretty pictures that we made using the tools, courtesy of Edwin (thanks!)

I hope everything will go all right. There will be another demonstration on friday, so plenty of opportunities to see the tools in action!

Open source repository and issue tracker online

2009-06-19T05:10:00.000-07:00

Yesterday, we reached an important milestone in the project. We have put a public repository online that contains the complete source code for all the tools. That's right: you can download the sources, tinker with them, and use them however you want.

We selected Assembla as our hosting for this project. It supports the Git distributed source code repository system, and nicely integrates that with an issue tracker. So far, it seems to be pretty flexible and works nicely. While the institute is developing her new website, we have put up a temporary website on the toolkit as well. The address will not change once the new site is up, so bookmark away!

If you are familiar with C++, or want to learn that: try your hand to help develop these tools. It's really not all that difficult. Of course, just reporting issues, suggesting documentation updates or giving ideas for improvements and extensions are also very valuable contributions!

Introducing: the ISI Data Importer

2009-06-19T04:53:00.000-07:00

This is the first of what is to become a series of postings to introduce all the tools in the toolkit. I hope it will give a clear overview of what kind of tools we offer, and what they do.

The ISI data importer is aimed at importing bibliographic data that you downloaded from ISI/Web of Knowledge. You can download data on the resulting articles from your searches in a text format. The ISI Data Importer tool can read these files and output them to a structured database format. The usage of structured databases is one of the basic ideas of the Scrience Research Toolkit. Using structured, standard databases to house the data allows us to use standard tools. Databases have been in development for decades, and are quite efficient for many tasks that suit the kind of work we do with the data. Also, getting the data in a form that is as structured as possible, gives us maximum flexibility.

The interface of the ISI Data Importer is quite simple:
On the first tab, you select the input file or files. You can select as many files as you want, as long as they are located in a single directory. As Web of Knowledge only allows you to download a maximum of 500 records in one go, you can end up with lots of separate files that all contain a fraction of your data. Simply select them all, and they will all be imported in a single run. Double records will automatically be filtered out, so if you have created several sets that can overlap in their results, you will end up with a single, unified set without double data points that can ruin your similarity measures later on.

On the output tab, you can select an output file. Currently the only supported database backend are Microsoft Access files, but we are working on extending that to include other and better database backends. Access can be a bit limiting and slow, especially if you work with large datasets. The filename you select does not need to exist yet. It will simply be created for you if it doesn't.

Optionally, you can filter the data on the document types. Some of the more frequently occurring document types are included in the list on the Filter page. If you are missing an option, let me know, and I'll add it. Better yet: simply patch the list yourself, the sources are available!

Demonstration on e-Social Science conference

2009-06-17T07:28:00.000-07:00

As announced in the introductory posting, we will be launching our toolkit on the e-Social Science conference in Colone, Germany. We will be doing that in a 20 minute demonstration session, where will we demonstrate how you can easily use a set of data downloaded from ISI/Web of Knowledge to create some maps of a research field, using a couple of database queries and our tools.

update, june 18:
~~As soon as I know the exact time, date and location of this session, I will post it here.~~
There will be two demonstration sessions:

11:00 – 11:30: Thursday 25 June
~~16:00 – 16:30: Thursday 25 June~~
13:40 – 14:00: Friday 26 June.

All demo's have been allocated to take place in the main foyer of Maternushaus.

If you happen to be at this conference, don't hesitate to join in for this demonstration!

First post

2009-06-12T02:20:00.001-07:00

Every blog needs an introductory posting, and this one is no different. What is it about? What can we expect to hear? Why even bother blogging? Those are the kinds of questions both you and I would like to see answered. "You and I", I hear you wonder? Yes, because it is not completely clear to me either what exactly I will and will not write about yet. So let's start with giving some idea about what I am doing, and why that is interesting to keep a weblog about.

The Science System Assessment department of the Rathenau Instituut is dealing, among other things, with trying to apply bibliometrics and patentometrics to map the dynamics of science and knowledge transfer. The problem that our department quickly ran into was that the available tools that can deal with this kind of data are relatively far between, require many manual, error prone and labour-intensive steps and don't fit together well. Worse, we soon ran into limitations with the amount of data we could handle in them that started to affect our research.

So, we decided to build some tools ourselves. Seeing that the tools that were (and are) available are not open, we had to start from scratch. That presented both a challenge and an opportunity, because in this way we could also rethink basic issues of how these tools should work. We decided to go for a design where all tools work against standard relational databases in which we structure the available data as well as possible. We also wanted the tools to be easy to use, so a clear graphical interface was a must. Since I have experience developing software using the excellent C++ based Qt toolkit, I chose to use that as the environment to build these tools in. As an added bonus, cross platform compatibility as well as database back end independence come practically for free.

As the first tools began to be available in early versions, more and more ideas about what else we could do and needed begun to pop up, and soon the idea to build some tools led to a complete toolkit that is still growing. Now the time has come to make these tools available to you too. The toolkit will officially be launched at the 5th conference of the National Centre for e-Social Science in Cologne. The first tree tools will be released in their "1.0" or "ready to use" versions, while the rest of them are made available "as is". Because we would have liked to contribute to the exisiting tools but could not, we have decided to avoid the same issue with our initiative.

We would love to hear from you, and even better, to work with to to improve these tools! We expressely invite you to use them, test them, and improve on them. To make that possible, we are making all sourcecode available under a liberal open source licence. We will also make a public issuetracker available, as well as a forum and other collaboration tools.

And that brings us to the why of this blog: we feel that it is important to keep you up to date with what is happening, what we are planning, and what others are doing with these tools. This blog is one of the ways in which you can do that. We are also working on a nice website, and a temporary website will be up soon. If you have other ideas about how to communicate, want to aggregate your own blog, or have any other comments: don't hessitate to contact me. I'd love to hear from you!