OpenSUSE install and some luck

Last weekend I  installed OpenSUSE 10.3. It was a bumpy installation, since I was without any CD I could burn. So I downloaded the ISO image to FAT32 partition, two additional files (kernel + initrd) and started install. I added another (non-free software) CD image to the list of repositories, but the install had failed afted being done with the 1st CD (Because the CD image was on the partition that was about to be added to /etc/fstab, with no formatting or anything destructive. But that was enough to confuse the installer and make it exit with an error.). So I tried to boot into the installed OS and.. the boot failed. So I was only left with a GRUB command line (because the menu.lst file had not been created) and a WinXP partition which I had to figure out how to boot from.

GRUB built-in help is the most useless help ever.

I  needed the chainloader command, ok, but that command wants "blocklist or file name". However, the blocklist syntax is nowhere to be found in the GRUB help. I was just about to go to work and burn the installation CD, when I saw another command (blocklist) which, given filename, prints out a list of blocks. I knew the kernel image filename on the root partition, so I finally got to see how the blocklist syntax is supposed to look like (ie. 0+1 = 1 sector starting from 0th). That enabled me to boot into XP, move stuff to unused partition and successfully install SUSE.

HOWEVER... I had a critical password stored on the root partition which got mkfs'd and overwritten several times.. and it was filled up to 60%, like never before. I thought that I had lost the password, but I tried my luck and used dd to make an image of the root partition into a file. Then I used the strings program to dump all printable strings in the partition image and, unbelievably enough, the password was there, not overwritten by the new data! WOW :) I had more luck than brains this time :)

As for SUSE, I'm running KDE now, and.. it's OK. Livable. Not bad. Portage broke down on my old gentoo installation and I finally got sick of it. Time to spend as little time managing my computer as possible.

Weekend in northern Sweden

I took a prolonged weekend (fri-mon; 30.11-03.12. 2007) in Northern Sweden; more accurately near the town of Piteå, visiting a friend. Actually, his house is on a nearby island that has only recently been connected to the mainlands with a road. The trip went (by plane) Oslo -> Stockholm -> Luleå (Kallax airbase) where he picked me up with a car. During those few days, we also visited some other places: Luleå, Kalix, and Haparanda/Tornio which lies on the border with Finland. (Haparanda is Swedish name, and Tornio is Finnish name.) You can see all of those places on the map. He bought some spirit drinks in Finland (the shop is named "Alko"), and I bought a bunch of Finnish salmiak/lakris candies. It was interesting to see people in Alko pushing FULL shopping carts of various drinks. Apparently, Haparanda has just recently gotten an IKEA shop, which has boosted the local economy. It is the nearest IKEA shop for that part of Sweden and Finland. Otherwise, it's been a very relaxing and interesting weekend, and they even got me to play bowling; the bowling hall was very nice for such a small town.

I was amazed by Swedish roads. Even though we were kinda in the middle of nowhere, the roads were in good shape (no holes or such) and very well cleansed - there was some snow/ice on them, but despite winter conditions it was possible to drive at speeds over 110 km/h (though with "piggdekk" - spiked tires). Norwegian roads are really bad compared to Swedish roads. (And I mean bad - even in Oslo you can find roads with holes at size of a football-ball in diameter.)

On the way back, I tried to take two cans of surstromming, but it was unfortunately taken by the security officer in Luleå. (BTW, he was really nice - told me that I should have checked that in, asked me whether I was coming back soon [he wanted to store it..]. It's rare to meet a nice security guy.. I guess that small towns have their advantages too :-)) BTW, there's also a technical university at Luleå.

Being far up north, near polar circle, I wanted to experience the "perpetual darkness". Well, it's not perpetual, but pretty close. So I have some pictures too. This picture is some kind of meeting place for young people in Piteå (the two arcs); the most interesting thing is that the picture has been taken ca. 15h.

This picture has been taken at the brightest time of day (though, cloudy one). It is a road that approaches the friend's house. It's a representative picture for most of the roads that we've been driving on.

The following picture shows the frozen sea. It's impressive. Later during the winter, the ice gets so thick that the easiest way to the town is to use a snow scooter and just cross the water.

This picture shows the entrance to the IKEA in Haparanda. I have taken it because the leftmost flag is that of the Sami people.
 
The last picture is mostly for amusement: it's an announcement in local newspapers that, due to maintenance of the electrrcal network, there will be a short power outage. But notice the time of day :)

Netbeans

Few days ago I started to write some Java code (I'm using the Choco constraint solver) and tried to use the Netbeans 5.5 IDE (and the version 6 should be out soon). I have to say that I was pleasantly surprised. It is simpler to use than Eclipse, and feels faster. Now I have started to use it for C and C++ development and I'm wondering like... WHY haven't I given it a chance earlier :)

IT Underground 2007

Yesterday late night (or rather, today early morning) I returned from the IT Underground conference where I have been an invited speaker. I gave a tutorial (as it later turned out, rather advanced) on the ELF file format and shared libraries. I also commented on some possible uses of GOT, eg. to plant an invalid address at a GOT entry in order to trace accesses to global variables. You can get the slides here. Otherwise, conference as a conference - not much to tell :) I attended some interesting lectures and met some new people, and experienced some nice surprises :)

I flew with SAS over Copenhagen. On my way to the conference, after having checked in, I saw that I had been transferred to a later flight, scheduled to leave Copenhagen at 18:30, instead of the earlier flight that was shown on the ticket and which should have departed at ca. 13h. The clerk at the SAS's desk explained that the earlier flight was canceled because it was scheduled to use Dash-8 plane type. SAS had a number of incidents with these airplane types within the last month, and has, for safety reasons, grounded all Dash-8 planes. They have announced to sue the Dash-8 producer, because they suffer enourmous money losses due to grounded planes.

Anyway, since I had more than 7 hours of waiting time, I decided to take a walk through Copenhagen's center. On the metro station I asked some Danish girls about which line to take to the center and overheard another girl asking the same (literally: "I just want to get to the center"). Somehow we ended up entering the metro at the same door and I started the conversation by asking (in English): "Do you also have a long time until the next flight?" She said yes, asked me where I was going and I replied to Warszawa. Then she asked me whether I was polish and I replied that I was Croatian, after which she said in serbo-croatian: "Ja sam iz Novog Sada." (translation: "I'm from Novi Sad"); Novi Sad is a 2nd largest city in Serbia. Then we both started to laugh, were talking much about everything [turned out that she graduated in Russian studies at university in Novi Sad and is now learning Polish language], took a walk through the city center, ate a lunch and took a coffee in a cafe afterwards. Then I headed back to the airport, and she stayed a bit longer in the city, since her plane was leaving 2 hours after mine. It was a very nicely spent time, and when I left for the airport, I realized that we never har introduced ourselves.. I never asked her about her name, nor did she ask me about mine... we were too involved in talk about, like, just about everything.

As for referring to the language as "serbo-croatian".. I'm aware that many Croatians will probably object to this name, but I don't give a sh*t. It seems that I and that girl have very similar views on the language; personally I consider "croatian" and "serbian" being two dialects of the same language that are being kept different for political reasons. I understand what she is saying, she understands what I am saying (except for few words of region-specific slang), so why make a fuss about it? I wrote "serbo-croatian" just to make it explicit that her spoken dialect was of the "serbian" flavor. Nevertheless, her dialect was much closer to the lanugage I'm used to hearing in Zagreb from Zagreb old-timers than the dialect of people coming from Hercegovina and that are considered to be speaking, ironically enough, "croatian" language. Bah, crappy politics. In elementary school (back then when Yugoslavia still existed), we were taught that there were no "croatian" or "serbian" language, but only "croato-serbian" or "serbo-croatian". As much as it is politically incorrect to say it (but hey! Croatia is supposed to be a democratic country with freedom of speech), I personally feel that an attempt to fully separate this (single) languages into two separate languages feels.. not wrong, but just a wasted effort.

It would take me too far to try to explain here the small differences in spoken language between the serbian and croatian dialects. Suffice it to say that the differences between spoken "croatian" and "serbian" languages/dialcets are far smaller than regional differences in the spoken norwegian language (eg. between Bergen - Oslo - Tromsø dialects).

How to raise SW patent hell

Today I got a nice idea on how to show that SW patents are either total crap or an idea that could put every major SW producer in trouble.

So the plan is this: find a good security engineer and a good patent lawyer. Make a patent titled something like "A method for covertly taking control over software". Make sure that the patent text covers buffer overflows. Next step: sue for patent violation every company whose software has at least one unpatched buffer overflow. I'm sure things would get pretty hilarious in the courts.

A critique of C++

The usenet article with message ID 13h30fshbtk53eb@corp.supernews.com (google groups has not yet picked it up, so I can't provide a direct link) analyzes in detail what is wrong with the current C++ language. Interesting read.

News and 3-d logic solver

I have set up a news feed so that you can track changes on my web site. Today's news: solver for the 3-D logic game.

DTrace and C++

Solaris DTrace isn't that C++ friendly. The SDT provider doesn't work with C++ programs. But I have fortunately managed to find a workaround. Not pretty, but it works.

OOXML vs. ODF

Do I care whether OOXML becomes standardized? I don't. Why - because I think that it's basically a mistake to standardize a format that conflates presentation and content. In that respect, standardizing ODF was also a mistake. So I don't pretty much care if another such format is standardized. After discussing the issue a bit over a coffe with a friend, I said that they should have better standardized LaTeX. It seems that I'm not the only one who thinks so.

Spam, viruses and addressbooks

How do you know that someone has added your otherwise unpublished email address to their addressbook(s)? Of course, you suddenly start getting spam! I strongly suspect that there exist viruses/trojans whose only task is to steal data from your addressbook and send it away to .. (someone).

Linus on volatile

I'm referring to this thread on kerneltrap.org, where Linus is cited to be saying:
- in other words, the *only* possible meaning for "volatile" is a purely
single-CPU meaning. And if you only have a single CPU involved in the
process, the "volatile" is by definition pointless (because even
without a volatile, the compiler is required to make the C code appear
consistent as far as a single CPU is concerned).

He's absolutely wrong in his statement here (namely, that volatile is "by definition" pointless for a single CPU). The C99 standard says that any access to volatile object is a side-effect. This does not mean that compiler optimizations are effectively disabled; rather it means that the compiler must generate memory access instruction instead of caching the value in the register. [This is because accesses to volatile objects may produce side-effects, so the read value may change between reads without an intervening store instruction.]

Now, consider the following simple code in a uniprocessor configuration:

while(!flag) ;

which simply waits for the flag to become true (e.g. set by an interrupt handler). If flag is not declared as volatile, the compiler might well generate an infinite loop, but if the flag is declared as volatile, the compiler must generate code that will check the physical memory location in every iteration and thus the flag change from an interrupt handler will be detected.

So, volatile is important even on uniprocessors whenever there is a possibility of executing asynchronous code (e.g. interrupts). And its semantics is defined well enough to prevent errors like the one I have described above.

Oh well, I don't really care nor shall I bother to comment on this on the kernel mailing list. It's his kernel, I use it only on my desktop and I don't really care what future impact this change will have (and it might have far-reaching consequences that are really hard to discover; as any bugs related to asynchronicity). I have said long time ago that I'd never consider Linux for "serious" applications (i.e. anything else than a cheap desktop), and such displays of blatant ignorance by its leading developer just make my stand firmer.

More python brokenness: distutils

So I tried to compile Mercurial on Solaris with Sun's native compiler. I set CC and CFLAGS environment variables, but, as documented in distutils manual, the CFLAGS set in the environment are added to some predefined set of flags. This set of flags was adjusted to gcc, and Sun's compiler broke (no wonder) on some of gcc's options. It seems that compiler options are hardcoded in distutils at some obscure place. "Great" work!

After ca. 30 mins of fiddling around and searching on google, I recompiled the module using gcc.

Concurrent shared memory allocator

I have recently adapted Solaris's libmtmalloc(3) to satisfy allocation requests from shared memory segments; it works both with POSIX and SYSV shared memory. You can get the code here. I knew about a similar project, but I have no idea how well does it scale with increasing number of concurrent processes.

Prex, DTrace and linux lameness

Adam Leventhal has in his blog exposed the lameness of the SystemTap team (e.g. they claim that they weren't inspired by DTrace at all, but output format of some of their utilities is identical to that of DTrace). He also took an amusing photo showing that users are not very happy with SystemTap.

But I want to point to another very useful tool that got completely shadowed by DTrace: TNF tracing. In short, it is "printf() debugging" on steroids; read the tracing(3TNF) man page for more details. The TNF logging probes must be manually inserted in the source code at places where printf() would be put, and they produce no output by default. When you run the program under the control of prex(1) program, you may choose to log the otuput of all probes, or select just a subset of them (they are named!). The trace is later analyzed with the tnfdump(1) program. The best thing is that prex works also on running programs; it can attach to them, collect data and detach at later point.

TNF has been an enormous help in finding and correcting complex, time-dependent problems that I'd have much harder time to resolve otherwise.

Solaris signal handling bug, part 2

My initial report has been confirmed to be a genuine bug. Even more interesting, the bug has been introduced in Solaris 8, and hasn't been noticed until few days ago. Wow :)

A possible signal handling bug in Solaris

I know that combining threads and signals is a bad idea, but I just couldn't resist the temptation. It seemed like a nice solution until I've hit what seems to be a possible signal handling bug: sometimes the signal is not delivered on the alternate signal stack, even though a thread has requested so. You can read the full description here. I wonder whether I'll get a reply. In the mean-time, I'm abandoning threads in favor of multiple processes operating over a large shared memory segment.

Summary

It's a very busy time. I've passed (with the best grade) my final exam in Norwegian, and since then I'm busy with coding my PhD project. Have been busy with installing Solaris on some machines and bringing it into usable state (by installing many small but important packages), upgrading the compilers to latest versions, learning the dbx debugger, fighting with compiler bugs and discovering subtle but important restrictions of lock-free data structures.

Writing code also involves learning much about the C++ boost libraries and discovering some important weaknesses of C++ (this time: operators new and delete are class members which is very inconvenient if you want to impose external allocator onto existing classes). But I was also impressed how good the Sun's C++ compiler is when it comes to aggressive optimizations and heavy inlining of C++ code; this is very important with heavy template metaprogramming.

Each of these topics is interesting enough to deserve its own post, so expect more posts in the upcoming days.

X11 ssh forwarding

It is common knowledge that ssh -X or ssh -Y forward X11 connections. However, what I didn't know until few days ago is that this works even across several machines. That is, if you do ssh -Y from A -> B -> C, then opening an X program on machine C will forward the X connection to the X server at machine A. Very convenient.

Windows Research Kernel

This is interesting: Microsoft gives out the NT kernel source for academic, non-profit use. Too bad I don't have much time to play with it a bit; I've always been impressed by the very NT kernel (which is itself VMS-ish). More details here. A related project is ProjectOZ, a platform (simulator) for teaching principles of operating systems.

Window managers

After being a long-time user of the fvwm2 window manager, I have switched to Ion. The author, incidentally, dislikes the current state of open-source software, linux (calling it the Idiot-box Linux), and gnome. I share many of his opinions and have subscribed to his blog. Very interesting read!

One end and one beginning

Today I have closed the old web site where I and a friend of mine have tried to build a community. Building a community is hard, takes effort, and consequently, time. And time is a pretty scarce resource for us. We have decided to take down the old web, and I have moved the content to my new personal web: http://zvrba.net

Digitally signing files

When you want to digitally sign a file, you can either make an embedded signature or a detached signature. The disadvantage of the embedded signature is that the original format is mangled. You have to "unpack" the file to another file (the process of verification does this) to be able to use it. Not very convenient. Detached signature preserves the original format, but stores the signature in a separate file. Unless you have a convention and hold firmly to it, you will have problems with pairing up signatures and their corresponding files. Modern filesystems open up a third possibility: a signature can be placed into a file's extended attribute. This solves both problems: the file's original format is preserved, and the signature is still attached to the file.

Algorithm toolbox

I have released a preliminary version of an algorithm toolbox. The toolbox includes van Emde-Boas trees, intrusive AVL trees (thus, applicable in scenarios where there are no memory management facilities), and exhaustive permutation tester. The code is written in C++ and may be obtained here.

Solaris 11, xemacs and dbx

I have recently installed Solaris Express (Solaris 11), first in one virtual machine and then on "bare metal". I have to say that I'm very pleasantly surprised. The installation went very smooth, the only "configuration" neccessary was to enter time, location, select additional locales, type in the root password and to create a new non-root user upon first login. The default desktop is GNOME and looks very polished up. Security-wise, the installation seems very sane. There are many services bound on TCP ports, but almost all of them only to the loopback interface (meaning that only local programs can connect to them). The only publicly open ports are 111 (rpcbind) and 22 (ssh), 6011 and 6010 (I guess, X). And sshd is configured to not permit root logins by default.

I tried Sun Studio 11, but.. being used to emacs, it doesn't feel quite right (+ it's not comfortable to use it if you're running it over remote X11 connection). However, Solaris has a very nice toolbox for programmers: dbx, mdb, dtrace and tnf (man tracing), different malloc libraries, etc. It's really very advanced OS, and very developer-friendly. Plus there are zones, fine-grained privilege and role system which works... And everything is excellently documented in man pages. I'm too short on time right now to describe all the goodies, but.. I'm impressed. By the installation, default settings, and overall consistency and look&feel of the system. I can only say excellent work to all the Solaris developers, and a big thanks to Sun for making it all available at no charge.

Now, to the topic: I had a problem with xemacs not displaying source files when I used it with dbx to debug a multi-directory project. The problem turned out to be that dbx does not by default display full path to the source file. xemacs looks for the file in the wrong directory and consequently does not show the source you're supposed to be debugging. You can solve this problem by putting the following line in your .dbxrc:

DBX_output_short_file_name=off

Linux signal handling is broken

Enter sigaction() in combination with SA_SIGINFO flag. Such signal handler accepts three arguments, the third being the context (full machine state needed to resume it, eg. registers) of the interrupted thread.

First problem: linux ABI is broken. The FP state in the uc_mcontext member of the ucontext_t structure is pointer, instead of value. This makes copying of the context nontrivial.

Second problem: You can't use setcontext() to leave signal handler and jump into another, previously saved, context. (Or, for that matter, you can't use it to return to the very same context passed as argument to the signal handler.) In other words, signal handler like

static void sighandler(
  int signo, siginfo_t *psi, void *pv)
{
  memcpy(puc_old, pv, sizeof(ucontext_t));
  /* choose another context to dispatch */
  setcontext(puc_another);
}

does not work. It does not restore signal mask specified in the puc_other, does not reestablish alternate signal stack, etc. However, this scheme works flawlessly on Solaris.

How am I fixing it on Linux? I walk the stack frames (following the saved stack frame pointer), modify the return address so that the signal handler returns to itself instead to the interrupted context, etc. Very ugly and nonportable.

Not to mention that I'm relying on luck: it seems that, under current linux kernel, it is not possible to atomically restore signal mask and return from signal handler to context other than the immediately interrupted one. (Heck, it's not even possible to do it nonatomically without resorting to "black magic" involving reading hex dumps of stack frames.)

I'm installing Solaris in a virtual machine to try it out, and I'm seriously considering to move my development to Solaris.

Three jewels

This one is, I guess, going to be another classic in a few years. Andrei Alexandrescu stated in comp.lang.c++.moderated:
Well, for my money, using iostreams already puts the code in a low quality bracket :o).

The thread was about how people are dogmatic regarding the goto statement, so here are two more links:

• D. E. Knuth, "Structured Programming with goto statements" [PDF]
• Frank Rubin's response [PDF] to the Dijkstra's paper on "harmful goto", published in readers's letters, Communications of the ACM, March 1987, Volume 30, Number 3.

Subverting the GPL (again!)

If you use some GPL code in your program and you distribute that program, than you also must make the program source available. A fun thing would be to comply with the GPL, but publish the source code of the parts of the program that you wish to keep "secret" in obfuscated form.

Here's a relatively easy way to do it: put "proprietary" parts of your code into separate source files, compile them to object code and decompile the object code back to "source". "Nice" decompilation is of course impossible. But we don't want nice - we can just generate code full of gotos, primitive machine instructions, and variable "names" which correspond to the variable's address in memory after the program is loaded.

I think that it would work, because the license states only "Accompany it with the complete corresponding machine-readable source code..." Such source code is undeniably machine-readable (and even compileable!), and it seems that distributing a "proprietary" program in such form would comply with the license.

Comments?

Another look on climate change

"This has been the warmest winter ever", "We must reduce CO2 at all costs", etc. You can hear such things pretty often today. But take another perspective: the global warming, which caused the warmest winter 'ever' (highly doubtful), also saved huge amounts of energy that would otherwise be spent on heating. Less energy used on heating means less generated CO2 this winter. Why doesn't anyone mention that?

Hungarian notation

You have probably heard that hungarian notation is awful and that it should be avoided. However, in most of the Windows APIs, the notation has been misused. The intention behind Simonyi's notation is akin to dimensional analysis. I recommend you to read this article as well as Simonyi's original paper.

While having a good intention behind it, the idea is actually just a fix for a defficient language, namely C and C++. If typedef were not a mere type alias, but a true new type, this convention would be (almost) unneccessary as the compiler could catch all errors. For example:

typedef int Temperature;
typedef int Volume;

int f(Temperature a, Volume b)
{
  return a+b;
}

Although it is meaningless to add temperature and volume quantities, this is perfectly legal C and C++ code, because Temperature and Volume are mere aliases for int.

C++ templates have strict compile-time type-checking, but the syntax is unfortunately just horrendous.

Giants meet

I am actually not surprised about this. I'm just wondering why didn't it happen sooner.

A thought on neural networks

Neural networks are a (currently not very successful) attempt to programmatically mimic the learning behaviour of a human brain. Research in neural networks has mostly focused on topologies and transfer functions in the nodes. However, the aspect of time has been neglected. Human nerves transport signals with (relatively slow) speed between 0.5 and 120 m/s (quoting a random reference from the web).

Could it be that our learning capability depends not only on particular signal values (the part that artificial neural networks are simulating), but also on propagation time between neurons in the brain? A signal might have different effect on a neuron in the brain, depending on the time of signal's arrival.

This would add another dimension to artificial neural networks: temporally changing transfer functions in the nodes. This is just an idea for further research, maybe someone has already looked into it.

x86-64 ISA oddity

New processors have FXSAVE and FXRSTOR to save/restore the floating point state of both x87 and SSE units. In 64-bit mode, PUSHAD and POPAD instructions generate undefined operation exception, there is no new instruction to save/restore all general-purpose registers, and the hardware task switching is disabled. I'm wondering why did the AMD designers decide to cripple the CPU in such a way (along the side of disabling segmentation, but that's a story for another post).

Enforcing mandatory exercise at work

I'm working in a large but low (only 4 floors) building with many printers. Today I came upon a slightly evil idea. Instead of letting people choose the printer on which to print their documents, the administrators should arrange the system to choose the destination printer randomly. Benefits would be twofold:

1. People would be forced to walk longer distances, which is healthy. This is beneficial especially for people working at computers, sitting most of their working hours.


2. I also guess that paper would be saved: in order to avoid the previous point, people would avoid printing stuff unless they absolutely have to.

To make the first point most effective, people should be locked out of elevators, except disabled people which would get their own "elevator key".

Assembler as a first "language" ?

I participated recently in a discussion where someone suggested to teach programming to beginners with assembler as the first language. There were far many more negative than positive reactions. My opinion is that assembler is just a tool to show concepts in practice. If used properly, it serves as well as any other tool (ie. high-level programming language). Here are Knuth's reasons for using assembler in his TAOCP books.

Now, I agree that it is much harder to make a good course with assembler - topics, examples and problems must be chosen much more carefully than for a high-level programming language. And x86 is a wrong choice as a first architecture - something like Knuth's MMIX or the MIPS architecture would be much better.

Bottom line is that, while there are admittedly many traps, I don't see anything fundamentally wrong with such approach. Your opinions?