I did a first attempt at creating a demonstration video for our SAINT tools. The first tool to get it, is the Matrix Builder tool. It's very basic, and runs for less than a minute and a half. I will create more elaborate videos later on, but it may be useful for somebody already.
Update: the video has been replaced with a higher resolution version. This should improve readability quite a lot. It may take a little while until the HD version is actually updated and you'll notice the improved quality.
vrijdag 15 januari 2010
Wrong file in download link
When uploading all the files, and updating the CMS to point to them, I made a mistake. The installer I file I pointed to from our CMS at the Rathenau Website was actually a very old installer. The installer pointed to from this blog was the correct one.
The mistake has been fixed now, but still: embarrassing. Sorry for any confusion or inconvenience!
The mistake has been fixed now, but still: embarrassing. Sorry for any confusion or inconvenience!
woensdag 13 januari 2010
Release 2010.01 beta 1 is out!
Yes, I finally made a new release ready for testing.
This new release fixes several bugs, the one bugging you most probably being the ISI parser issue. Even though there was a work-around, it was an annoying issue. Sorry about that!
Another big thing in this release is that it is the first release to include the community detection. As part of the Network Tools program, you can now employ the algorithm described in V. Blondel, J.-L. Guillaume, R. Lambiotte and E. Lefebvre 2008, Fast unfolding of community hierarchies in large networks. More on that algorithm you can find on their website.
As already described in the last blog posting, there are also some changes under the hood of the Matrix Builder tool. Besides labels and edge values, you can now also output other properties to the Pajek .net output file, allowing you to display more aspects at the same time in your network.
There are many other small tweaks and bugfixes in this version. Expect more additions before the final release. When that will be released? When it is done!
Download the latest beta toolkit (installer for Microsoft Windows) here.
This new release fixes several bugs, the one bugging you most probably being the ISI parser issue. Even though there was a work-around, it was an annoying issue. Sorry about that!
Another big thing in this release is that it is the first release to include the community detection. As part of the Network Tools program, you can now employ the algorithm described in V. Blondel, J.-L. Guillaume, R. Lambiotte and E. Lefebvre 2008, Fast unfolding of community hierarchies in large networks. More on that algorithm you can find on their website.
As already described in the last blog posting, there are also some changes under the hood of the Matrix Builder tool. Besides labels and edge values, you can now also output other properties to the Pajek .net output file, allowing you to display more aspects at the same time in your network.
There are many other small tweaks and bugfixes in this version. Expect more additions before the final release. When that will be released? When it is done!
Download the latest beta toolkit (installer for Microsoft Windows) here.
woensdag 9 december 2009
Big changes in Matrix Builder
More news on updated tools: the Matrix Builder.
While simple in concept, the Matrix Builder is a very important tool in our toolbox. And it has seen some important changes! Let me take you through them.
Just like the ISI parser, the matrix compiler has had some subtle interface changes. The window can now be minimized, and the close button has been removed. The old OK button has been renamed to Run. This will result in less confusion.
More importantly, is the introduction of properties. That is: more data can be attached to the output than just connection strengths and vertex labels. There are many such properties possible, and we have a wish list for almost all of them. The basics are implemented, and all that remains now is to implement more 'plugins' to support more properties.
Quite a few nice properties have already been implemented. The most useful will probably be the vertex size property (scale node sizes according to some value), and the vertex colouring.
The new version will bear version number 1.3
While simple in concept, the Matrix Builder is a very important tool in our toolbox. And it has seen some important changes! Let me take you through them.
Just like the ISI parser, the matrix compiler has had some subtle interface changes. The window can now be minimized, and the close button has been removed. The old OK button has been renamed to Run. This will result in less confusion.
More importantly, is the introduction of properties. That is: more data can be attached to the output than just connection strengths and vertex labels. There are many such properties possible, and we have a wish list for almost all of them. The basics are implemented, and all that remains now is to implement more 'plugins' to support more properties.
Quite a few nice properties have already been implemented. The most useful will probably be the vertex size property (scale node sizes according to some value), and the vertex colouring.
The new version will bear version number 1.3
ISI parser updates
It has been a long time since my last blog. That does not mean that developments have stopped! I'll try to fill you in on what happened in the meantime. I'll start with the oldest tool in the box: the ISI data importer.
Recently, I made an update for the ISI data importer. Most changes are pretty small, but still...
The first issue that was addressed, is a limitation that existed with selecting files. If you selected lots of files, especially if they had long file names in a deep level of you directory hierarchy, it could happen that you'd run over 32 thousand characters for the file names. That resulted in leaving some files out, without any warning! The issue was addressed by changing the way multiple selected files are displayed in the file selection widget. Instead of just listing all the file names including their paths (which you will never read anyway for 50+ files), you now get a listing like
Another issue that was addressed, is the laggy display of the progress. If you hid or obscured the progress window during a parsing operation, it would take untill the next new file untill it was updated again. This is now fixed. What's more, the window can be minimized, and the main window is hidden during the parsing process. The progress bars are also changed. The files progress bar has been replaced by a write progress bar, that displays the amount of parsed records (articles) being written to the database.
A new feature has also been introduced: it is now possible to add data to an existing database! That means that if you select an existing database as your output file, you are now forced to choose what to do with that. You can either augment the existing data (no duplicates will be made) or you can overwrite the existing database completely.
A user interface change will enforce that you actually have make a choice: The OK button has been renamed in a Run button, and the Close button has been removed. Closing is done by just closing the window. The Run button will only be available if no problems have been detected. If there are problems that prevent running, hovering your mouse over a small warning-sign icon will tell you about what they are.
This new version of the ISI data importer will get version number 1.2.
Recently, I made an update for the ISI data importer. Most changes are pretty small, but still...
The first issue that was addressed, is a limitation that existed with selecting files. If you selected lots of files, especially if they had long file names in a deep level of you directory hierarchy, it could happen that you'd run over 32 thousand characters for the file names. That resulted in leaving some files out, without any warning! The issue was addressed by changing the way multiple selected files are displayed in the file selection widget. Instead of just listing all the file names including their paths (which you will never read anyway for 50+ files), you now get a listing like
'file_1' and 999 other files in directory ''file_0.txt' and 999 other files in directory 'C:/Documents and Settings/andre/Desktop/data'Much more readable. The way the display is formatted, depends on the number of files selected.
Another issue that was addressed, is the laggy display of the progress. If you hid or obscured the progress window during a parsing operation, it would take untill the next new file untill it was updated again. This is now fixed. What's more, the window can be minimized, and the main window is hidden during the parsing process. The progress bars are also changed. The files progress bar has been replaced by a write progress bar, that displays the amount of parsed records (articles) being written to the database.
A new feature has also been introduced: it is now possible to add data to an existing database! That means that if you select an existing database as your output file, you are now forced to choose what to do with that. You can either augment the existing data (no duplicates will be made) or you can overwrite the existing database completely.
A user interface change will enforce that you actually have make a choice: The OK button has been renamed in a Run button, and the Close button has been removed. Closing is done by just closing the window. The Run button will only be available if no problems have been detected. If there are problems that prevent running, hovering your mouse over a small warning-sign icon will tell you about what they are.
This new version of the ISI data importer will get version number 1.2.
donderdag 6 augustus 2009
Progress with Relation Calculator
The relation calculator tool that I introduced in my last blog, is shaping up nicely. Sure, it is not all smooth sailing, but it is really starting to look like something that could be very useful indeed. Let's use a screenshot of the current version to illustrate where things are heading:
What do you see?
The main thing you'll notice is the area on the right, where you see boxes that are connected by lines. Each one of these boxes represents one simple step in the process of doing an analysis. In this case: calculating a Jaccard coefficient based on a co-word/cited reference combinations between journal papers. A nice, basic selection of such basic steps - components in the terminology of the application - is already there, though not fully implemented yet. The list of components can be extended later on using plugins. The available components are visible in the list on the top left.
Each of the components have inputs on their left and/or outputs on their right. Outputs can be connected to one or more inputs for other components, thus creating a graph. Note that no circular connections are allowed. The user can use drag & drop to put components on the screen, and to connect inputs and outputs together.
So, how do these separate components become an analysis?
On the bottom left of the screen, you can see the Execution order window. Here you can see the order in which the components will be executed. This is determined by their connections, the question if they present an interface at run time, and their positions on the screen. You can open and save analysis sequences for reuse and distribution.
Once you have hooked up every component in the right order, you can run it. You will be presented with a wizard-type interface that guides you through the UI elements that each of the components presents (if any), and that present information about the progress of your analysis if there are long run times involved.
What do you see?
The main thing you'll notice is the area on the right, where you see boxes that are connected by lines. Each one of these boxes represents one simple step in the process of doing an analysis. In this case: calculating a Jaccard coefficient based on a co-word/cited reference combinations between journal papers. A nice, basic selection of such basic steps - components in the terminology of the application - is already there, though not fully implemented yet. The list of components can be extended later on using plugins. The available components are visible in the list on the top left.
Each of the components have inputs on their left and/or outputs on their right. Outputs can be connected to one or more inputs for other components, thus creating a graph. Note that no circular connections are allowed. The user can use drag & drop to put components on the screen, and to connect inputs and outputs together.
So, how do these separate components become an analysis?
On the bottom left of the screen, you can see the Execution order window. Here you can see the order in which the components will be executed. This is determined by their connections, the question if they present an interface at run time, and their positions on the screen. You can open and save analysis sequences for reuse and distribution.
Once you have hooked up every component in the right order, you can run it. You will be presented with a wizard-type interface that guides you through the UI elements that each of the components presents (if any), and that present information about the progress of your analysis if there are long run times involved.
vrijdag 17 juli 2009
Building a new tool: Relation Calculator
As announced in the previous posting about development prioritites, I will be working on a tool to calculate scores for relations between objects in the database. That sounds very general, and it actually is. Allow me to elaborate...
background
The basic way of creating relations in our relational-database based data store is to create SQL queries that express the relation (e.g. correlation or distance measure) you are after. You can for instance relatively easily express a bibliometric coupling measure in SQL. It is also possible, but already more complicated, to express a Jaccard index over, say, title word similarities between articles in your database.
This approach, while very flexible in theory, still has some limitations in actual practice. The first is user-related: not every researcher wanting to do this kinds of analysis is a hero in creating SQL queries. That limits the usefulness of the tool set, not because what people want can not be done, but because it has a too high learning curve to actually do so. Providing standard database views for standard analyzes only solves this to a limited level.
Another limitation is that database engines are not always as efficient as they could be in evaluating the expressions that you need to construct often used measures. Also, they tend to use only a single thread to do a single query, thus making limited use of the resources of our modern multi-core computers. Using GPGPU techniques to speed up calculations is completely out of the scope of SQL for the foreseeable future. All this means that our calculations take a lot longer then they need to take, and sometimes run into arbitrary limits that they need not run into.
As stated above, providing standard views can only work up to a point. We want to be able to calculate relationships between all kinds of items in the database (articles, journals, authors, ...), and we want to be able to use different measures for them as well (Jaccard, cosine, Salton, ...). Providing standard solutions for all of them is simply not doable. It would result in an exponential increase of pre-defined views for everything you want to add, and would basically create a mess in what is embedded in the database. That's not a very inviting prospect. What we need is something that is flexible enough to allow calculating relations between all kinds of items in all kinds of ways we can think of.
So, we need a tool that can make these calculations:
The Relation Calculator tool should become that tool for you. It uses a concept of building blocks, that can be put together to create a calculation that makes sense. Each building block will be responsible for a small part of the chain, like selecting which database to operate on, selecting views and fields, loading the data from the database, calculating a Jaccard index, etc. Each building block has inputs and/or outputs, that can be connected to each other. In this way, a calculation for a relation can be defined. Some building blocks will present a UI to the user during execution of the calculation, for instance to allow selecting a database or some parameter like a threshold. Other building blocks will just perform some service, or maybe even one simple logical operation. Of course, these configurations of blocks can be loaded and saved, to be re-used later on. A set of pre-defined configurations can then be presented to the user.
New building blocks can be added as plugins later on, making the tool extensible. Another option to extend the functionality is to use building blocks that execute a script as their payload. For instance, you would be able to define a function in JavaScript that expresses a relationship between two authors based on some input data. That script can then be used in a configuration. The possibilities are virtually endless.
I envision a graphical environment where a user would be able to drag and drop building blocks on a canvas and graphically connect the input and outputs of the blocks. This would create a very simple way to define new configurations to calculate new relations.
relation to existing code
There already is some code that does the kind of work I described here. Like I mentioned in my previous blog, the current Record Grouper basically calculates such relations already. There is also already some code available that lays the basis for a script-based relation calculator. These existing pieces will of course not be just thrown away. They will have to be refactored to be able to use them as building blocks in the new Relation Calculator.
status
I am now very bussy implementing the infrastructure for all this. Though it is a lot of work, I am confident that it will work. Some details still have to be filled in, but I don't expect major obstacles in the near future. I hope to get a basic model working soon.
The idea is that
background
The basic way of creating relations in our relational-database based data store is to create SQL queries that express the relation (e.g. correlation or distance measure) you are after. You can for instance relatively easily express a bibliometric coupling measure in SQL. It is also possible, but already more complicated, to express a Jaccard index over, say, title word similarities between articles in your database.
This approach, while very flexible in theory, still has some limitations in actual practice. The first is user-related: not every researcher wanting to do this kinds of analysis is a hero in creating SQL queries. That limits the usefulness of the tool set, not because what people want can not be done, but because it has a too high learning curve to actually do so. Providing standard database views for standard analyzes only solves this to a limited level.
Another limitation is that database engines are not always as efficient as they could be in evaluating the expressions that you need to construct often used measures. Also, they tend to use only a single thread to do a single query, thus making limited use of the resources of our modern multi-core computers. Using GPGPU techniques to speed up calculations is completely out of the scope of SQL for the foreseeable future. All this means that our calculations take a lot longer then they need to take, and sometimes run into arbitrary limits that they need not run into.
As stated above, providing standard views can only work up to a point. We want to be able to calculate relationships between all kinds of items in the database (articles, journals, authors, ...), and we want to be able to use different measures for them as well (Jaccard, cosine, Salton, ...). Providing standard solutions for all of them is simply not doable. It would result in an exponential increase of pre-defined views for everything you want to add, and would basically create a mess in what is embedded in the database. That's not a very inviting prospect. What we need is something that is flexible enough to allow calculating relations between all kinds of items in all kinds of ways we can think of.
So, we need a tool that can make these calculations:
- more easy to use (at least for the standard analyzes), and
- faster to compute, while still
- be as flexible as possible.
The Relation Calculator tool should become that tool for you. It uses a concept of building blocks, that can be put together to create a calculation that makes sense. Each building block will be responsible for a small part of the chain, like selecting which database to operate on, selecting views and fields, loading the data from the database, calculating a Jaccard index, etc. Each building block has inputs and/or outputs, that can be connected to each other. In this way, a calculation for a relation can be defined. Some building blocks will present a UI to the user during execution of the calculation, for instance to allow selecting a database or some parameter like a threshold. Other building blocks will just perform some service, or maybe even one simple logical operation. Of course, these configurations of blocks can be loaded and saved, to be re-used later on. A set of pre-defined configurations can then be presented to the user.
New building blocks can be added as plugins later on, making the tool extensible. Another option to extend the functionality is to use building blocks that execute a script as their payload. For instance, you would be able to define a function in JavaScript that expresses a relationship between two authors based on some input data. That script can then be used in a configuration. The possibilities are virtually endless.
I envision a graphical environment where a user would be able to drag and drop building blocks on a canvas and graphically connect the input and outputs of the blocks. This would create a very simple way to define new configurations to calculate new relations.
relation to existing code
There already is some code that does the kind of work I described here. Like I mentioned in my previous blog, the current Record Grouper basically calculates such relations already. There is also already some code available that lays the basis for a script-based relation calculator. These existing pieces will of course not be just thrown away. They will have to be refactored to be able to use them as building blocks in the new Relation Calculator.
status
I am now very bussy implementing the infrastructure for all this. Though it is a lot of work, I am confident that it will work. Some details still have to be filled in, but I don't expect major obstacles in the near future. I hope to get a basic model working soon.
The idea is that
Abonneren op:
Posts (Atom)